WorksComputer Science
Razavi-Bench: Tracking AI Progress in Analog Circuit Reasoning
Abstract
We propose Razavi-Bench, a benchmark built from the 50 questions in Razavi's two Analog Design Experiments With AI articles and graded by GPT-6 Luna against reference answers and rubrics, and use it to track how fast AI models improve at analog circuit analysis. Across 73 multimodal large language models released between May 2024 and November 2026, the top score rose from 28.83% to 98.33% in 30 months, mostly before the reference answers were published. The leading models now answer almost every textbook question correctly; the hardest questions show a circuit that resembles a textbook one but works differently, or ask about behavior the obvious analysis misses. Within two and a half years, a set of questions that exposed the limits of AI models has become one they answer almost perfectly.
Project Page: https://razavi-bench.tokenzhang.com
Repository: https://github.com/Arcadia-1/razavi-bench
Can AI Models Analyze Analog Circuits?
How well can an AI model reason about analog circuits? Razavi’s two Analog Design Experiments With AI articles asked whether a model can interpret a schematic, identify the mechanisms at work, and predict the effect of a design change. ChatGPT answered fewer than half of his undergraduate-level analysis questions correctly [6], and Gemini 3.1 Pro just over half of his design-oriented ones [7]. Both misread devices or signal paths, or applied valid formulas under the wrong assumptions.
These are two snapshots of a fast-moving field. Chip-design benchmarks such as VerilogEval [3] and RTLLM [4] score the digital code a model writes, and AnalogCoder [2] the analog circuits it designs. Others test circuit understanding: CIRCUIT [9] grades the numerical answers to 510 analog-circuit questions, AMSbench [8] asks about 8,000 multimodal questions on schematic perception, analysis, and design, and MMCircuitEval [10] 3,614 on digital and analog circuits across the design flow, while CircuChain [5] pairs circuit-analysis problems with variants that invert the usual sign conventions to test whether models follow the stated conventions or their training priors. Razavi-Bench complements them in three ways: its 50 questions were written by an analog designer, Razavi, who also graded AI answers to them, and we calibrate our reference answers and grader against his verdicts; it follows 73 models released over 30 months on the same questions; and its look-alike circuits separate models that analyze a drawn circuit from those that recognize its shape. Razavi-Bench turns the 50 questions into a benchmark and applies it to 73 models (Figure 1). Four findings stand out:
Figure 1. Overall scores of all 73 models at their highest reasoning effort, by release date. The dashed step line is the top score available at each date; the dotted line marks June 30, 2026, when the full reference answers became public.
The top score rose from 28.83% to 98.33% in 30 months, mostly before the reference answers were published (§ 3).
Textbook circuits are nearly solved; look-alikes are not. The hardest questions show a circuit that resembles a familiar one but works differently, or ask about behavior the obvious analysis misses (§ 4).
Models identify a circuit before they analyze it. The strongest models then check the individual devices; most others keep the familiar name (§ 4.2).
The last points are the expensive ones, but the cost of reaching 90% fell 17-fold within five months (§ 5).
Benchmark and Evaluation Setup
Each model receives one message holding the question text and, where needed, the schematic, cropped to the part the question asks about and without any title that would name the circuit. There is no system prompt and no tool or web access. Models run at the highest reasoning effort each offers and at its maximum output length, and answer every question in three independent trials. The Overall score is the share of available rubric points awarded. Cost covers all 150 answers at OpenRouter’s list prices on November 10, 2026, excluding grading.
GPT-6 Luna, at maximum reasoning effort, grades each answer against the reference answer and rubric [11], from 0 points (incorrect) to 4 (correct conclusion with correct reasoning). Its grades agree closely with Razavi’s own on the AI answers he graded in the two articles: its totals match his for Part 1 and fall within three points for Part 2. The agreement is partly by construction, since we refined the references using his verdicts; with those removed, its totals fall a few points below his. All questions, rubrics, answers, and grades are public on the project page.
Progress Over Time
The first questions were published in November 2025, and the full reference answers on June 30, 2026. About 40% of the improvement came before the first of these dates and nearly 90% before the second, so most of the gain cannot have come from training on the reference answers.
Ten models now score at least 90%. The three leaders answer nearly every question fully correctly and never score zero. The top two, Gemini 4 Carbon and Claude Opus 5.5, are within trial-to-trial variation of each other (their trials span 98.0% to 98.5% and 97.0% to 98.5%), and both are clearly ahead of the rest: the worst trial of each beats every other model’s best.
Release date correlates with score, but recent models still vary widely, largely with price: among models released since July 2026, the median score rises from about 65% for those costing under $1 for the 150 answers to about 90% for those costing more than $10.
Which Questions Are Hard, and Why?
The easiest and hardest questions
Table 1 lists the easiest and hardest questions. The easiest ask about single-device facts or textbook concepts, which even the oldest models mostly answer correctly. Circuits that must be recognized from the drawing were hard until recently: no 2024 model recognized a plain source follower, whereas nearly every recent model analyzes it fully.
| Key | Question | Mean score | Full credit |
| Easiest | |||
| P1 Q2 | Two students disagree on whether a MOSFET’s transconductance rises or falls with overdrive voltage; who is right? (Both, depending on what is held constant) | 99.0% | 96% |
| P1 Q14 | Analyze the CMOS inverter of Figure 2(c) (open-ended) | 97.5% | 95% |
| P1 Q3 | Is a PMOS device’s small-signal model identical to an NMOS device’s? | 97.3% | 93% |
| P1 Q1 | What happens to a MOSFET’s intrinsic gain if its length and width double? | 93.7% | 85% |
| P1 Q22 | Explain why the Miller effect is less pronounced in a cascode | 92.2% | 72% |
| Hardest | |||
| P1 Q21 | Is the circuit of Figure 2(b) a cascode? (No: a PMOS source follower drives a common-gate stage) | 23.1% | 21% |
| P1 Q15 | Analyze the two-NMOS circuit of Figure 2(d) (open-ended) | 26.3% | 24% |
| P2 Q14 | Role of an inverter added across the master latch of a divide-by-two circuit (a feedforward path, not a keeper) | 31.6% | 9% |
| P2 Q15 | Is that added inverter a keeper, as a quoted answer claims? (No) | 33.2% | 7% |
| P2 Q5 | Small-signal resistance looking into a ring oscillator’s supply node | 33.7% | 15% |
| P2 Q19 | How a capacitor added at an LC oscillator’s tail node changes phase noise | 37.7% | 2% |
| P2 Q6 | Does the supply-node resistance of P2 Q5 change if the ring oscillates? (Yes) | 38.8% | 5% |
| P1 Q6 | Can a PMOS device with gate and drain grounded act as a current source? (No: it is diode-connected) | 39.4% | 27% |
Table 1. The easiest and hardest questions, by mean score over the 219 answers to each (73 models × 3 trials). P1 and P2 denote Part 1 and Part 2.
The hardest questions fall into two groups. The first shows a circuit or device that resembles a familiar one but works differently: the CMOS inverter and the cascode redrawn with a changed transistor type (§ 4.2), a diode-connected transistor offered as a current source, and an inverter in a divide-by-two circuit that looks like a latch keeper but actually forms a feedforward path. Models tend to answer the familiar circuit rather than the drawn one; even Claude Fable 5 reads the keeper in every trial.
The second group asks how a circuit behaves where the obvious analysis does not apply. Part 2 questions 5 and 6 ask for the small-signal resistance looking into the supply node of a ring oscillator. Treating each inverter as two diode-connected devices gives , but the ring oscillates, and the resistance is then set by the charging of the node capacitances, about . Most answers give only the static result, and even when the next question states that result and asks whether oscillation changes it, few answers get it fully right. A capacitor added at an LC oscillator’s tail node (Part 2 question 19) can raise phase noise in one range of values, lower it at another, and cause low-frequency instability when it is too large. Most answers give only one direction, usually that the capacitor helps.
The leader’s few remaining errors are mostly tradeoffs, such as widening a comparator’s cross-coupled pair, which speeds the latch up only until the added capacitance takes over. It makes no errors on the look-alikes.
Part 2 is harder on average than Part 1, yet models that answer without extended reasoning score higher on Part 2. Part 2 asks how performance changes, which a model can partly recall from textbooks, whereas Part 1 mostly asks it to work out how a drawn circuit operates, and that is where reasoning pays off.
Recognizing or analyzing?
Does a model analyze a circuit, or recognize its shape and recall the familiar one? Two pairs of Part 1 questions tell these apart (Figure 2). Question 21 shows the cascode of question 20, except that the lower transistor is a PMOS device with its drain grounded, so it acts as a source follower driving the common-gate transistor. Question 15 shows the CMOS inverter of question 14 with two NMOS devices, so the upper one is a source follower.
Figure 2. Two textbook circuits (green) and their look-alikes (red), from Part 1 questions 20, 21, 14, and 15, with each question’s mean score and, as read by GPT-6 Luna, how the answers identify the circuit.
Answers that call a look-alike by its textbook name always score zero, and those that see through it average more than three of the four points. Newer models fall for the look-alikes less often, but many still do; the top models almost never do.
Reasoning traces show the order of the steps. Most first name a familiar circuit, often before stating any transistor type, even when the question does not name it. For textbook circuits this shortcut works; on the look-alikes the outcome depends on whether the model later tests the name against the symbols. The failures take two forms. On the cascode look-alike, about half the failing answers never discuss the transistor types; on the inverter look-alike, most failing traces work out the transistor types from the symbols and still keep the familiar name, and even those that later drop it recover only partial credit. Two traces show the difference:
Claude Fable 5.2 (4 points): “Now I’m checking M1’s arrow at its top terminal, which marks it as a PMOS with its source at the top, tied to M2’s source, its drain grounded, and its gate at Vin. That makes M1 a source follower feeding M2’s source, so the stack is not a cascode.”
Claude Opus 4.8 (0 points): “Checking the transistor symbols, the arrows don’t settle the NMOS question, but the topology is clear regardless. . . . That’s the textbook cascode configuration, so the answer is yes.”
The strongest models check consistently and get both look-alikes right in every trial. Below them, seeing through one look-alike does not mean seeing through the other. Because many traces are provider summaries, which may compress the order of the steps, this ordering is indicative rather than exact; the full, unsummarized traces, mostly from mid-ranked models, show it even more starkly.
What Does Cost Buy?
Figure 3 plots score against cost. Along its Pareto front, each step up costs more than the last: about 88% costs under $3 for the 150 answers, about 95% about $8, and the last 3.5 points to the top score another $20.
Figure 3. Overall score against the API cost of all 150 answers at OpenRouter’s list prices (logarithmic axis). The dashed line is the Pareto front; colors are as in Figure 1.
Gemini 4 Carbon pays for its lead in tokens, writing about 3.5 times as many per answer as Gemini 4 Argon at the same price per token. The most expensive models, recent Claude Fable and Opus versions, cost more than twice as much as Gemini 4 Carbon without beating it. Cost and score correlate, even among recent models.
A correct answer has become only somewhat cheaper. On the questions answered with full credit both before 2026 and since July 2026, the median full-credit answer costs $0.039 from the recent models against $0.052 from the older ones, because cheaper models now answer correctly; within each question, the cost of a full-credit answer has risen, as answers grew longer faster than tokens became cheaper. What has fallen sharply is the cost of reaching a given score (Table 2). Nearly two-thirds of the questions can now be answered reliably for about a cent each; money matters only on the rest.
| Score | Cheapest when first reached | Cheapest in November 2026 | Reduction |
| ≥60% | Gemini 3 Flash, Dec 2025, $2.41 | Qwen 3.7 Flash, Jul 2026, $0.22 | 11× | |
| ≥70% | Gemini 3 Flash, Dec 2025, $2.41 | GPT-6 Luna, Sep 2026, $0.93 | 2.6× | |
| ≥80% | GPT-5.3, Feb 2026, $19.27 | Gemini 3.7 Flash, Aug 2026, $2.73 | 7.1× | |
| ≥90% | Claude Fable 5, Jun 2026, $79.33 | Hy4, Nov 2026, $4.77 | 17× |
Table 2. The cheapest model reaching each Overall score, in the month the score was first reached and in November 2026. Costs are for all 150 answers.
Conclusion
The errors that remain are of two kinds. Many models answer from a circuit’s familiar shape, and outside the strongest models, checking the name against the transistor symbols is inconsistent: a model that sees through one look-alike can fall for another. Separately, the leader’s remaining losses are on tradeoffs that have no one-directional answer. Benchmarks that aim to measure circuit analysis rather than recall should therefore include both look-alike circuits and such tradeoffs on purpose.
Razavi-Bench also has clear limits: its 50 questions are few, short, and self-contained, its leader is within two points of the maximum, and its grades come from a single language-model judge. Analog design means turning a specification into a circuit, simulating it, and revising it; tracking progress from here will need agentic design benchmarks such as Analog Design Bench ([1]) and, beyond them, tasks that are harder, more varied, and closer to industrial design practice.
References
- [1]Analog Design Bench Team. Long-Horizon Analog Design Bench: Benchmarking agents on hours-long analog and mixed-signal circuit design tasks. arXiv preprint arXiv:2609.33356, 2026. URL https://arxiv.org/abs/2609.33356.
- [2]Yao Lai, Sungyoung Lee, Guojin Chen, Souradip Poddar, Mengkang Hu, David Z. Pan, and Ping Luo. Analog-Coder: Analog circuit design via training-free code generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 379–387, 2025. doi: 10.1609/aaai.v39i1.32016.DOI
- [3]Mingjie Liu, Nathaniel Pinckney, Brucek Khailany, and Haoxing Ren. VerilogEval: Evaluating large language models for Verilog code generation. In IEEE/ACM International Conference on Computer-Aided Design (ICCAD), pp. 1–8, 2023. doi: 10.1109/ICCAD57390.2023.10323812.DOI
- [4]Yao Lu, Shang Liu, Qijun Zhang, and Zhiyao Xie. RTLLM: An open-source benchmark for design RTL generation with large language model. In Asia and South Pacific Design Automation Conference (ASP-DAC), pp. 722–727, 2024. doi: 10.1109/ASP-DAC58780.2024.10473904.DOI
- [5]Mayank Ravishankara. CircuChain: Disentangling competence and compliance in LLM circuit analysis. arXiv preprint arXiv:2602.15037, 2026. URL https://arxiv.org/abs/2602.15037.
- [6]Behzad Razavi. Analog design experiments with AI—Part 1 [The Analog Mind]. IEEE Solid-State Circuits Magazine, 17(4):11–15, 2025. doi: 10.1109/MSSC.2025.3611213.DOI
- [7]Behzad Razavi. Analog design experiments with AI—Part 2 [The Analog Mind]. IEEE Solid-State Circuits Magazine, 18(2):8–13, 2026. doi: 10.1109/MSSC.2026.3686589.DOI
- [8]Yichen Shi, Ze Zhang, Hongyang Wang, Zhuofu Tao, Zhongyi Li, Bingyu Chen, Yaxin Wang, Zhen Huang, Xuhua Liu, Quan Chen, Zhiping Yu, Ting-Jung Lin, and Lei He. AMSBench: A comprehensive benchmark for evaluating MLLM capabilities in AMS circuits. arXiv preprint arXiv:2505.24138, 2025. URL https://arxiv.org/abs/2505.24138.
- [9]Leila Skelic, Yan Xu, Matthew Cox, Wenjie Lu, Tao Yu, and Ruonan Han. CIRCUIT: A benchmark for circuit interpretation and reasoning capabilities of LLMs. arXiv preprint arXiv:2502.07980, 2025. URL https://arxiv.org/abs/2502.07980.
- [10]Chenchen Zhao, Zhengyuan Shi, Xiangyu Wen, et al. MMCircuitEval: A comprehensive multimodal circuit-focused benchmark for evaluating LLMs. arXiv preprint arXiv:2507.19525, 2025. URL https://arxiv.org/abs/2507.19525.
- [11]Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems, volume 36, pp. 46595–46623, 2023.arxiv.org/abs/2306.05685


