betaFeedback

WorksComputer Science

MedJev: A System One Vision-Language Decision Model for Fast and Effective Medical Image Understanding

Abstract

Medical image understanding with vision-language models (VLMs) typically relies on autoregressively generating reasoning and answer tokens. These unnecessary tokens make the inference cost scale with the response length and force the answer to be parsed from free-form text, incurring avoidable latency and decision errors. To this end, we introduce MedJev, a system one vision-language decision model that maps medical images, questions, and candidate answers directly to decisions in a single multimodal forward pass, without any autoregressive generation process. For fast inference, MedJev removes the upper language layers and the generative output pathway, while keeping the full vision pathway and all visual tokens. On this shortened backbone, a decision head scores each candidate by pretrained query–key matching and refines the scores with a lightweight multilayer perceptron (MLP) residual, coupling the pretrained matching structure with task-specific nonlinear discrimination. Built on Qwen3.5-4B, MedJev-4B improves both accuracy and speed over the original model on each of four medical image classification and visual question answering benchmarks, with an absolute accuracy gain of 7.93% and a \textbf{6.04×\times} end-to-end speedup. Further comparison with 20 cutting-edge models shows that MedJev-4B achieves state-of-the-art performance and inference speed among 4B–9B general and medical VLMs, indicating that direct multimodal decision making is an effective route to fast and effective medical image understanding.

§ Code: https://github.com/zfkarl/MedJev

Introduction

Vision-language models (VLMs) have become a general interface for understanding medical images. Both general-purpose VLMs [3, 4, 24, 43, 58] and medical VLMs trained on biomedical image-text data [5, 20, 34, 48] can read dermoscopy images, radiographs, and biomedical figures and answer clinical questions about them in natural language. Going one step further, recent medical VLMs are trained to write out their reasoning traces before giving the final answer [17, 29]. For all of these models, obtaining the result of a medical image task relies on autoregressive generation and decoding.

This autoregressive process is an expensive and indirect way to obtain these results. Each generated token requires another forward pass through the language model, so the inference cost grows with the length of the response (Figure 1 (b)), and reasoning traces make the response much longer. Even with thinking disabled, the original Qwen3.5-4B [31] answers a typical question of the multi-image benchmark MedThinkVQA [52] in 0.35 s in our measurements, yet its 95th-percentile latency exceeds 20 s and its mean latency reaches 1.84 s. Beyond this cost, the decision has to be parsed from free-form text with rules or an extra model, and a response that matches none of the candidates is counted as an error (Figure 1 (a)). Moreover, the reasoning traces contribute little, as more than 70% of model errors on MedThinkVQA come from reading the images and integrating evidence across views, and and longer traces help little once this grounding is weak [52].

Figure 1. (a) An example from PMC-VQA [55], where Qwen3.5-4B writes 1,024 tokens without selecting an option and MedJev-4B scores the candidates in one forward pass. (b) Per-request latency against generated tokens, thinking off. (c) Average accuracy and latency on four benchmarks, where MedJev-4B beats all 20 general and medical VLMs at the 4B–9B scale on both.

The above two problems both stem from producing the decision as generated text. Dual-process theories of cognition describe two modes of thinking, a fast and intuitive system one and a slow and deliberate system two [9, 15]. Inspired by this theory, Jev models [2] have quickly become popular, mainly because they give up free-form generation and return typed and probabilistic decisions directly, which removes both the decoding cost and the output parsing. Many medical image tasks ask for one choice among a fixed set of diagnoses or answers, which matches this form of decision making. We therefore aim to understand medical images in a system one manner and propose MedJev-4B, a fast and effective vision-language decision model built on Qwen3.5-4B. Given one or more medical images, a question, and a set of candidate answers, MedJev appends a marker token to each candidate and a decision token to the end of the input, scores all candidates jointly in a single forward pass, and takes the highest-scoring candidate as its prediction. Since no reasoning or answer tokens are generated, the decision cost no longer grows with the response length, and a prediction is always valid.

The core idea of MedJev is to make the decision with less language-side computation while keeping all visual evidence and the matching structure learned in pretraining. It realizes this idea with two complementary designs, a language-side depth reduction that shortens the path from the visual evidence to the decision for speed, and a residual decision head that refines pretrained query–key matching for accuracy on the heterogeneous medical decisions.

Language-Side Depth Reduction. Starting from a pretrained VLM, MedJev keeps the vision encoder, the vision-language connector, and every visual token that they produce. Since studies of large language models find that many of the deepest layers can be removed with limited loss in performance [10, 27], it removes the language-side computation that the decision does not use, namely the upper layers before the last one, the value, gate, output, and feed-forward paths of the last layer, and the vocabulary projection. In MedJev-4B, these components account for 20% of the parameters. Visual token pruning methods [6, 47, 56] accelerate VLMs by discarding encoded visual tokens, which may remove small lesions and fine structures. MedJev therefore keeps the visual evidence intact and shortens only the language-side path to the decision.

Residual Decision Head. To score the candidates, MedJev reuses the query and key projections of the last layer, which already learned in pretraining how strongly one token attends to another. It maps the decision token to a query and each candidate marker to a key, and takes their scaled dot product as the matching score of the candidate. Because pretrained matching alone may not be expressive enough for heterogeneous medical images and questions, a lightweight multilayer perceptron (MLP) shared across candidates adds a residual score computed from the decision state, the candidate state, and their elementwise product. Its output layer is initialized to zero, so training starts from pretrained matching and learns task-specific nonlinear discrimination with few new parameters.

Extensive experiments on four medical image understanding benchmarks that cover dermoscopy, chest radiographs, multi-image clinical cases, and biomedical figures [39, 50, 52, 55] demonstrate the superiority of MedJev-4B over the competing baselines. Compared with the original Qwen3.5-4B, MedJev-4B improves the average accuracy by an absolute 7.93%, from 61.13% to 69.06%, and accelerates end-to-end inference by 6.04×, from 685.03 ms to 113.44 ms, with accuracy gains on every benchmark. These gains need no task-specific output head, because a single model scores candidate sets that range from a binary chest radiograph decision to a seven-way dermoscopic diagnosis. The speedup tracks what generation would have cost, so it peaks on the multi-image benchmark, whose cases drive a generative model to the longest responses of the four. In addition, comparison with 20 off-the-shelf general and medical VLMs (Figure 1 (c)) shows that MedJev-4B is the only model at the 4B–9B scale that is both the most accurate and the fastest on average.

The main contribution of this paper is threefold:

  • We introduce MedJev, a system one vision-language decision model that maps medical images, questions, and candidate answers directly to decisions in a single forward pass, without any autoregressive generation.

  • We design a language-side depth reduction that removes the computation a decision does not need while keeping every visual token, and a residual decision head that refines pretrained query–key matching with a lightweight zero-initialized MLP for accurate decisions.

  • On four medical image understanding benchmarks, MedJev-4B improves the average accuracy of Qwen3.5-4B by an absolute 7.93% with a 6.04× end-to-end speedup, and achieves state-of-the-art accuracy and speed among 4B–9B general-purpose and medical VLMs.

Related Work

Medical Vision-Language Models

Medical VLMs largely inherit the generative design of general-purpose VLMs and adapt it to clinical data. Early models tune LLaVA [24] on biomedical figure-caption pairs [20], learn from interleaved medical images and text [28], or train one generalist model across many medical tasks and modalities [40, 45]. Later models scale up instruction data and backbones [5, 34, 48], while strong general-purpose VLMs [3, 4, 19, 43] have also become competitive baselines on medical tasks. Inspired by chain-of-thought prompting [44] and reinforcement learning with verifiable rewards [11], recent work further trains medical VLMs to write out reasoning steps before the answer span [17, 29]. These models are evaluated on benchmarks that range from a mix of closed and open questions [12, 18, 22] to multiple-choice questions over given candidates [14, 52, 55], yet all of them deliver their decisions as generated text. Because the decision arrives inside a sentence, these benchmarks recover it by matching strings or by asking a separate judge model, and a response that states the right finding in an unexpected form can still be scored wrong. Contrastive models such as BiomedCLIP [54] instead score an image against text labels directly, but they encode the two modalities separately and cannot condition the scores on a question or on several images. MedJev combines both families by keeping the joint multimodal processing of a VLM while replacing text generation with a direct decision.

Efficient Vision-Language Inference

The inference cost of a VLM comes mainly from long visual token sequences, the depth of the language model, and autoregressive decoding. Token reduction methods discard or merge visual tokens [6, 35, 47, 51, 56], which shortens the input but risks removing small medical findings. Layer pruning methods remove redundant deep layers of large language models [10, 27], yet the pruned models still generate their answers token by token. Decoding also leaves the cost of a single answer open, since a model may spend a few tokens or several hundred on the same question. A generated answer carries the decision inside open-ended text, so the decision shares the parameters and the computation of captioning and dialogue. System one models [2] instead remove decoding and return typed decisions, and concurrent work extends them to images [53, 57], videos [59], spatial reasoning [30], histopathology [38], and ophthalmic cases [23]. Among them, MJev-Spatial [30] reads candidate scores from the native query and key projections at the top of the full language stack, while earlier work scores option letters with the language model head [33]. MedJev removes decoding and the upper language layers together, keeps every visual token, and and refines the pretrained query–key matching with a zero-initialized residual head.

Methodology

MedJev turns a pretrained VLM into a decision model that reads the images, the question, and all candidate answers in one pass and returns a score for each candidate (Figure 2). It formulates medical image understanding as a selection among the given candidates (§ 3.1), shortens the language side of the backbone to the computation that this selection needs (§ 3.2), and scores the candidates with a residual decision head built on pretrained query–key matching (§ 3.3). The whole model is trained with a single cross-entropy objective and runs with no decoding loop and no parser (§ 3.4), and Algorithm 1 summarizes the full training and inference procedure.

Figure 2. Overview of MedJev. (a) The images, the question, and all candidates form a single prompt with learned markers, and every visual token is kept. (b) A reduced backbone returns the marker states. (c) Pretrained query–key matching and a shared MLP residual score the candidates in one pass. The example is a CT question of PMC-VQA [55] with the probabilities of MedJev-4B.

Decision Formulation

Each example contains M≥1M \ge1 medical images I={I1,…,IM}\mathcal{I} = \{\mathcal{I}_1, \ldots, \mathcal{I}_M\}, a question qq, and KK candidate answers C={c1,…,cK}C = \{c_1, \ldots, c_K\}, and the goal is to predict the index y∈{1,…,K}y \in\{1, \ldots, K\} of the correct candidate. The candidates are written in text and can be diagnostic labels, findings, or answer options, so classification and multiple-choice VQA share the same form. A generative VLM solves this task indirectly. It produces a response o=(o1,…,oT)\mathbf{o} = (o_1, \ldots, o_T) token by token:

p(o∣I,q,C)=∏t=1Tp(ot∣o<t,I,q,C),(1)p(\mathbf{o} \mid\mathcal{I}, q, \mathcal{C}) = \prod_{t=1}^{T} p(o_t \mid o_{<t}, \mathcal{I}, q, \mathcal{C}), \tag*{(1)}

and a parser then maps o\mathbf{o} to a candidate, so the cost grows with TT and the prediction fails whenever o\mathbf{o} names no candidate. MedJev instead computes a score vector s=fθ(I,q,C)∈RK\mathbf{s} = f_{\theta}(\mathcal{I}, q, \mathcal{C}) \in\mathbb{R}^{K} in a single forward pass and predicts:

y^=arg⁡max⁡k∈{1,…,K} sk,(2)\hat{y} = \underset{k \in\{1,\ldots,K\}}{\arg\max}\ s_k, \tag*{(2)}

which is always a given candidate.

To compute s\mathbf{s} in one pass, MedJev writes the question and all candidates into a single prompt in the chat template of the backbone and adds three special tokens to its vocabulary. A question marker ⟨q⟩\langle q\rangle follows the question, an option marker ⟨o⟩\langle o\rangle follows each candidate, and a decision marker ⟨d⟩\langle d\rangle closes the prompt after an instruction to select the best answer, so that the language model receives:

[V1,…,VM,q,⟨q⟩,c1,⟨o⟩,…,cK,⟨o⟩,⟨d⟩],(3)[V_1, \ldots, V_M, q, \langle q\rangle, c_1, \langle o\rangle, \ldots, c_K, \langle o\rangle, \langle d\rangle], \tag*{(3)}

where VmV_m denotes the visual tokens of image Im\mathcal{I}_m and each candidate is preceded by its option letter. The marker embeddings are initialized from the end-of-turn token of the backbone and learned during training. Let tkt_k be the position of the option marker after ckc_k and tdt_d the position of the decision marker. Under causal attention, the hidden state at tkt_k summarizes the images, the question, and the candidates up to ckc_k, while the hidden state at tdt_d summarizes the entire input. These marker states are the only outputs that the decision requires from the language model.

Language-Side Depth Reduction

A generative VLM runs its full language stack and the vocabulary projection at every decoding step, whereas a decision only needs the hidden states at the marker positions. MedJev exploits this asymmetry. A probe of the frozen backbone confirms this, since the marker states stop becoming more discriminative well before the last layer (Figure 3). Consider a backbone whose vision encoder and vision-language connector map each image to visual tokens, and whose language model stacks LL layers F1,…,FLF_1,\ldots,F_L with an attention layer at the top. MedJev keeps the vision encoder, the connector, and the first JJ language layers unchanged and removes layers J+1J+1 to L−1L-1, so the states that reach the decision head are:

Figure 3. Depth probe analysis. Balanced accuracy of the pretrained query-key match combined with a linear head over the marker states of each pretrained layer, with the backbone frozen and all examples from the training split.

ht=[(FJ∘⋯∘F1)(X)]t,t∈{t1,…,tK,td},(4)\mathbf{h}_t = \left[(F_J \circ\cdots\circ F_1)(\mathbf{X})\right]_t,\qquad t \in\{t_1,\ldots,t_K,t_d\}, \tag*{(4)}

where X\mathbf{X} is the embedded input sequence of Eq. (3) and [⋅]t[\cdot]_t takes the DD-dimensional state at position tt. From the last layer FLF_L, MedJev keeps only the input normalization, the query and key projections with their normalizations, and the rotary position embedding [36], which the decision head reuses.

The value and output projections, any attention gate, and the feed-forward network of FLF_L are removed together with the final normalization and the vocabulary projection, so the language model executes J+1J+1 modules instead of LL. Every visual token is kept, and none is pruned or merged inside the language model, so the reduction shortens only the language-side path to the decision.

Residual Decision Head

The last layer was pretrained to measure how strongly one token attends to another, which matches the relation that a decision needs between its marker and each candidate. MedJev therefore scores the candidates with the retained query and key projections. Let N(⋅)N(\cdot) be the input normalization of the last layer, and let hˉd=N(htd)\bar{\mathbf{h}}_d = N(\mathbf{h}_{t_d}) and hˉk=N(htk)\bar{\mathbf{h}}_k = N(\mathbf{h}_{t_k}). For each query head a∈{1,…,H}a \in\{1,\ldots,H\}, the head computes:

q(a)=RtdNQ(WQ(a)hˉd),kk(a)=RtkNK(WK(g)hˉk),(5)\mathbf{q}^{(a)} = R_{t_d}N_Q\left(\mathbf{W}_Q^{(a)}\bar{\mathbf{h}}_d\right),\quad\mathbf{k}_k^{(a)} = R_{t_k}N_K\left(\mathbf{W}_K^{(g)}\bar{\mathbf{h}}_k\right), \tag*{(5)}

where WQ(a)\mathbf{W}_Q^{(a)} and WK(g)\mathbf{W}_K^{(g)} are the pretrained query and key projections, g=g(a)g = g(a) is the key head shared by query head aa under grouped-query attention [1], NQN_Q and NKN_K are the query and key normalizations, and RtR_t applies the rotary embedding at position tt. The matching score of candidate kk averages the scaled dot products over heads:

skQK=1H∑a=1H⟨q(a),kk(a)⟩dh,(6)s_k^{\mathrm{QK}} = \frac{1}{H}\sum_{a=1}^{H}\frac{\left\langle\mathbf{q}^{(a)},\mathbf{k}_k^{(a)}\right\rangle}{\sqrt{d_h}}, \tag*{(6)}

where dhd_h is the head dimension. This score is the attention logit of the last layer and is free.

Pretrained matching alone is not sufficient in this setting. The query and key projections were trained on the output of the penultimate layer, yet after depth reduction they receive the output of layer JJ, and a bilinear score may not capture every interaction that a medical decision requires. MedJev therefore adds a nonlinear residual computed by a lightweight MLP shared across candidates:

skMLP=w2TGELU⁡(W1zk+b1)+b2,(7)s_k^{\mathrm{MLP}} = \mathbf{w}_2^{\mathsf{T}}\operatorname{GELU}\left(\mathbf{W}_1\mathbf{z}_k+\mathbf{b}_1\right)+b_2, \tag*{(7)}

where zk∈R3D\mathbf{z}_k \in\mathbb{R}^{3D} concatenates hˉd\bar{\mathbf{h}}_d, hˉk\bar{\mathbf{h}}_k, and their elementwise product hˉd⊙hˉk\bar{\mathbf{h}}_d \odot\bar{\mathbf{h}}_k, and W1\mathbf{W}_1 projects it to a small hidden dimension. The final score of candidate kk is:

sk=skQK+skMLP.(8)s_k = s_k^{\mathrm{QK}} + s_k^{\mathrm{MLP}}. \tag*{(8)}

We initialize w2\mathbf{w}_2 and b2b_2 to zero, so that sk=skQKs_k = s_k^{\mathrm{QK}} at the start of training and the MLP learns only the correction that pretrained matching misses. Because all candidates are scored with the same parameters, the decision head is not tied to a fixed label set and handles any number of candidates.

Training and Inference

A single MedJev model is trained on a mixture of medical classification and VQA sources (Algorithm 1). For each example, the final scores are normalized with a softmax:

pθ(k∣I,q,C)=exp⁡(sk)∑j=1Kexp⁡(sj),(9)p_\theta(k \mid\mathcal{I},q,C) = \frac{\exp(s_k)}{\sum_{j=1}^{K}\exp(s_j)}, \tag*{(9)}

and the model minimizes the weighted cross-entropy:

L=−∑nwnlog⁡pθ(yn∣In,qn,Cn),(10)\mathcal{L} = -\sum_n w_n \log p_\theta(y_n \mid\mathcal{I}_n,q_n,C_n), \tag*{(10)}

Algorithm 1 Training and Inference of MedJev

Input: Pretrained VLM with language layers F1,…,FLF_1,\ldots,F_L, depth JJ, training set D={(In,qn,Cn,yn)}\mathcal{D}=\{(I_n,q_n,\mathcal{C}_n,y_n)\} with weights wnw_n

Output: Prediction y^\hat{y} for a test input (I,q,C)(I,q,\mathcal{C})

1: Add the markers ⟨q⟩\langle q\rangle, ⟨o⟩\langle o\rangle, ⟨d⟩\langle d\rangle to the vocabulary ▹ Sec. 3.1 2: Remove FJ+1,…,FL−1F_{J+1},\ldots,F_{L-1} ▹ Sec. 3.2 3: Reduce FLF_L to its query and key branches 4: Add a residual MLP with zero-initialized output ▹ Sec. 3.3 5: function fθ(I,q,C)f_\theta(I,q,\mathcal{C}) 6: X←\quad\mathbf{X}\leftarrow embedded input sequence of Eq. (3) 7: ht←[(FJ∘⋯∘F1)(X)]t\quad\mathbf{h}_t\leftarrow[(F_J\circ\cdots\circ F_1)(\mathbf{X})]_t for t∈{t1,…,tK,td}t\in\{t_1,\ldots,t_K,t_d\} 8: \quad return s\mathbf{s} with sk=skQK+skMLPs_k=s_k^{\mathrm{QK}}+s_k^{\mathrm{MLP}} ▹ Eqs. (5) to (8) 9: end function 10: for each minibatch B⊂D\mathcal{B}\subset\mathcal{D} do ▹ Training 11: sn←fθ(In,qn,Cn)\quad\mathbf{s}_n\leftarrow f_\theta(I_n,q_n,\mathcal{C}_n) for n∈Bn\in\mathcal{B} 12: LB←−∑n∈Bwnlog⁡softmax⁡(sn)yn\quad\mathcal{L}_{\mathcal{B}}\leftarrow-\sum_{n\in\mathcal{B}}w_n\log\operatorname{softmax}(\mathbf{s}_n)_{y_n} ▹ Eq. (10) 13: \quad Update LoRA, markers, norms of FLF_L, and MLP with ∇LB\nabla\mathcal{L}_{\mathcal{B}} 14: end for 15: Merge LoRA and markers into the weights ▹ Deployment 16: return y^=arg⁡max⁡kfθ(I,q,C)k\hat{y}=\arg\max_k f_\theta(I,q,\mathcal{C})_k ▹ Inference

where nn indexes the training examples and wnw_n compensates for two imbalances in the mixture. The sources differ greatly in size, so wnw_n gives every source the same total weight. Medical classification sources are also skewed toward frequent classes, so within such a source this weight is split equally among classes, while a VQA source counts as a single class because its candidates vary across questions. When minimizing L\mathcal{L}, we keep the pretrained backbone largely intact by freezing the vision encoder and applying LoRA [13] to the vision-language connector and to all linear layers of the retained language model, including the reused query and key projections. In contrast, the marker embeddings, the retained normalizations of the last layer, and the residual MLP are either newly introduced or small, so all of their parameters are updated directly in training.

At deployment, the LoRA updates are merged into the pretrained weights and the marker embeddings into the embedding table, so the deployed model has exactly the reduced architecture of Sec. 3.2 and no adapter overhead. A prediction then takes one forward pass over the input of Eq. (3) followed by Eqs. (5) to (8) and the selection in Eq. (2). Since no token is generated, it needs no decoding loop, no cache of generated tokens, and no answer parser. As a result, the latency depends on the input and the reduced depth alone, and every prediction carries a distribution over the candidates from Eq. (9).

Experiments

Experimental Setup

Benchmarks

We conduct experiments on four medical image understanding benchmarks that differ in imaging modality and in decision type. HAM10000 [39] asks for one of seven dermoscopic diagnosis categories and provides 1,512 test examples. PneumoniaMNIST [50] asks whether a pediatric chest radiograph is normal or shows pneumonia and provides 624 test examples. MedThinkVQA [52] presents a clinical case with 2 to 47 images and five candidate diagnoses and provides 720 test examples. PMC-VQA [55] asks a question about a figure from the biomedical literature with four options and provides 33,430 test examples. The four benchmarks together contain 36,286 test examples, and the number candidate answers per example ranges from two to seven. HAM10000 and PneumoniaMNIST label an image with a fixed set of diagnoses, while MedThinkVQA and PMC-VQA ask a question about each case, so the mixture covers classification over a fixed label set and question answering over free text.

Baselines

We compare MedJev-4B with 20 off-the-shelf vision-language models at the 4B to 9B scale. Fifteen of them are general-purpose models from the LLaVA [19, 25, 26], InternVL [7, 43, 58], Keye-VL [16, 49], MiMo-VL [46], and Qwen [3, 4, 32] families, and four are medical models trained on biomedical image-text data [5, 20, 34, 48]. The remaining one is the original Qwen3.5-4B [31], which we compare against most closely because it shares the vision encoder, the connector, and the retained language layers with MedJev-4B. Every baseline runs off the shelf with its official weights and chat template, and a response naming no candidate counts as an error. We restrict the comparison to the 4B to 9B range so that every baseline stays close to MedJev-4B in size.

Implementation Details

We build MedJev-4B on Qwen3.5-4B, whose language model has L=32L = 32 layers with a full-attention last layer. We keep the first J=23J = 23 layers, which reduces the parameter count from 4.44B to 3.55B, and the residual MLP has a hidden dimension of 256 and adds 1.97M parameters for D=2560D = 2560. In Qwen3.5, the query projection also produces an output gate, which we discard. We train a single model for one epoch on the 174,360 training examples of the four benchmarks with LoRA of rank 16 and scaling 32, which amounts to 5,449 updates with seed 42. We measure the end-to-end latency of every model on an NVIDIA H20 GPU at batch size one, including image loading, preprocessing, and model execution. Each baseline is timed in a separate session under the same protocol, and the latency of Qwen3.5-4B comes from the session that also times MedJev-4B, so that the original backbone and our model are measured under exactly the same timing conditions.

Main Results

As shown in Table 1, MedJev-4B is the most accurate model on all four benchmarks and the fastest on three of them and on average, whereas no baseline reaches both ends of that trade-off, with the most accurate one spending 5,579.8 ms on a case and the fastest one answering 21.79% of them correctly. The comparison that carries weight is Qwen3.5-4B, which shares the vision encoder, the connector and the first JJ language layers with our model, and against it we gain 7.93% of average accuracy while deciding 6.04× faster. Removing the decoding loop produces both gains, since a generated answer spends computation on tokens the decision never uses, and it spends more the longer the answer runs. Generative latency therefore spans two orders of magnitude across the four benchmarks, and MedThinkVQA shows it most clearly, where Qwen3.5-9B spends 20,947.7 ms on a case to reach the second highest average accuracy and MedJev-4B spends 250.7 ms. Free text also fails in two ways that a decision cannot. The multi-image cases of MedThinkVQA overflow the context window of LLaVA-Med-v1.5-7B and LLaVA-1.6-Vicuna-7B and leave nothing to read, and on HAM10000 the first of those two answers with the option letter inside a sentence that the fixed rule set of the protocol will not accept. A MedJev-4B prediction is a candidate index, so the parsing failure cannot occur, and the fixed visual budget keeps every case here inside the input window.

MethodHAM10000 Acc.↑HAM10000 Lat.↓PneumoniaMNIST Acc.↑PneumoniaMNIST Lat.↓MedThinkVQA Acc.↑MedThinkVQA Lat.↓PMC-VQA Acc.↑PMC-VQA Lat.↓Avg. Acc.↑Avg. Lat.↓
General Vision-Language Models
LLaVA-1.5-7B [25]7.41151.937.50140.06.81144.635.43144.321.79145.2
LLaVA-1.6-Vicuna-7B [26]3.90371.037.50211.80.00139.239.43257.620.21244.9
LLaVA-1.6-Mistral-7B [26]4.10408.237.34243.019.722,810.741.03290.725.55938.2
LLaVA-OneVision-7B [19]40.48435.240.38234.135.14909.356.22251.143.06457.4
InternVL2.5-4B [7]53.31801.451.44172.729.58765.351.05578.146.34579.4
InternVL3-9B [58]62.171,404.251.12247.932.08892.554.07940.149.86871.2
InternVL3.5-4B [43]59.261,001.858.33106.538.47836.159.06532.453.78619.2
InternVL3.5-8B [43]65.411,032.351.28132.032.50750.760.66677.452.46648.1
Keye-VL-8B-Preview [16]61.24151.039.74127.727.781,124.256.09128.246.21382.8
Keye-VL-1.5-8B [49]58.07160.838.46120.626.67615.955.15130.744.59257.0
MiMo-VL-7B-RL [46]31.48371.750.32320.224.178,806.549.811,353.038.952,712.9
Qwen2.5-VL-7B [4]23.88159.339.42100.332.22740.153.27113.637.20278.4
Qwen3-VL-4B [3]53.17112.666.51124.825.97365.251.93113.149.39178.9
Qwen3-VL-8B [3]59.66135.360.10106.830.42489.755.46128.151.41215.0
Qwen3.5-9B [32]64.95209.985.42196.740.6920,947.761.19964.963.065,579.8
Medical Vision-Language Models
LLaVA-Med-v1.5-7B [20]0.00505.518.91397.10.00231.71.30840.45.05493.7
HuatuoGPT-Vision-7B [5]26.52249.385.42161.835.97818.652.89170.150.20350.0
MedGemma-4B [34]56.48215.239.10207.831.94949.948.44213.043.99396.5
Lingshu-7B [48]60.25184.982.53126.839.03771.157.45141.759.81306.1
Qwen3.5-4B [31]58.40257.392.79267.336.391,842.856.95372.761.13685.0
MedJev-4B (Ours)68.3973.394.2364.851.94250.761.6865.069.06113.4
Improvement+9.993.51×+1.444.13×+15.567.35×+4.735.74×+7.936.04×

Table 1. Main results on four medical benchmarks with accuracy (Acc., %) and mean end-to-end latency (Lat., ms). Avg. is the unweighted mean and Improvement the gain over Qwen3.5-4B. Every row was measured on one H20 at batch size one, and we bold the best and underline the second best.

Further Analysis

Ablation Study

Figure 4 (a) and Figure 4 (b) vary the retained depth and the decision head and hold everything else fixed, with the accuracy axis zoomed so that every bar carries its value. Full keeps all 32 language layers and DR reduces them to J=23J = 23, QKH scores a candidate by pretrained query-key matching alone, MLP replaces that matching with a per-candidate MLP, and RDH adds the zero-initialized residual we deploy. The depth reduction costs no accuracy under either head, raising the mean from 68.31% to 68.70% under QKH and from 68.93% to 69.06% under RDH while cutting about 16 ms of latency. The depth probe of Figure 3 anticipates this, since a linear decision is already readable at layer 23 and the layers above serve the vocabulary head. The head comparison is closer. QKH alone reaches 68.70%, so the pretrained attention geometry carries most of the decision, and the MLP scorer adds little at 68.83%. RDH reaches the best mean of 69.06% for 1.97M added parameters and 1.4 ms over QKH, refining the matching score that MLP simply replaces.

Figure 4. Ablation and significance. (a) Unweighted mean accuracy and (b) mean end-to-end latency over the four benchmarks, where DR is the depth reduction, QKH the query-key head, MLP the per-candidate MLP and RDH the residual decision head. (c) The accuracy gain of MedJev-4B over the generative Qwen3.5-4B, with its 95% interval from a paired bootstrap of 10,000 resamples.

Significance of the Gains

In Figure 4 (c), we resample the test examples 10,000 times and recompute both accuracies, which turns each gain of Table 1 into a paired bootstrap interval. Three of the four stay clear of zero, so the gains of 9.99% on HAM10000, 15.56% on MedThinkVQA and 4.73% on PMC-VQA exceed the sampling noise of their test sets. The exception is the 1.44% on PneumoniaMNIST, spanning [−0.80, 3.69], and the narrowest interval belongs to PMC-VQA, so separation follows test set size as much as gain size. The intervals rest on disagreements and not on shared successes, since MedJev-4B alone is correct on 5,615 PMC-VQA questions while Qwen3.5-4B alone is correct on 4,035, and the split on MedThinkVQA is 221 against 109, so the gains rest on decisions that actually flip.

Candidate Order

In Figure 5 (a) and Figure 5 (b), we decide every test example again with the candidate order permuted under three seeds, because a model that scores candidates could be reading the slot an option occupies instead of its text. The permuted accuracies sit on the native bars and the unweighted mean falls only from 69.06% to 68.10%, so the order is not what the head reads. Panel (b) locates the small remainder, since a fixed label vocabulary loses nothing on PneumoniaMNIST and 1.6% on HAM10000 while free-form options lose 13.1% on MedThinkVQA and 9.4% on PMC-VQA.

Figure 5. Candidate order and hidden representations. (a) Accuracy under the native and three permuted orders. (b) The fate of a correct decision, kept or lost. (c, d) t-SNE of the decision marker before and after training and of the trained candidates, with the marginal distribution of each axis.

Hidden Representations

In Figure 5 (c) and Figure 5 (d), we reduce the states the head reads on 597 training examples with t-SNE and draw the marginal distribution of both axes. Before training the decision markers break into clusters, where two examples of one benchmark sit 2.36 times closer than two examples of different benchmarks, and training collapses them into one region at a ratio of 1.36, so one head serves candidate sets that range from two to seven answers. The ratio compares the mean distance between two examples of one benchmark with the mean distance between two examples of different benchmarks, measured on the same sample in the reduced space. The candidate markers separate in the same geometry, since the correct ones concentrate while the others spread out, which the matching term of Eq. (6) reads, and why QKH on its own already decides well.

Comparison with Released Decision Models

In Table 2, we run six released system one models through the entry point each release documents, two medical and four general-purpose, and give each the candidate texts the benchmark defines. Every model is scored from the first image of each case, which keeps the input identical across interfaces, and the latency column covers the three single-image benchmarks, as in the main table. MedJev-4B is the most accurate on three of the four and leads the mean by 11.20%, and only the 3B model is faster. Scale does not explain the ordering, since the two 9B models reach 56.57% and 57.82%. The medical pair is the narrowest, since OmniMed-Jev-4B takes PneumoniaMNIST by 0.80% but falls to 32.93% on PMC-VQA, while PathJev-P averages only 44.86%, below the general-purpose d3-flash-9B everywhere.

ModelHAMPneu.MedThinkPMCAvg.↑Lat.↓
d1-3B [21]59.9237.5031.5355.4346.0935.6
Visual Jev-4B [53]56.8162.0220.5651.9947.8478.5
OmniMed-Jev-4B [37]66.6795.0322.2232.9354.21201.4
PathJev-P [38]51.3264.1026.8137.2344.86192.1
Clef-Flash-9B [8]61.9079.9728.0656.3756.5783.1
d3-flash-9B [42]67.0071.4732.9259.8857.8275.9
MedJev-4B (Ours)68.3994.2351.8161.6669.0256.8

Table 2. Comparison with released decision models. Accuracy (%) on the four benchmarks, Avg. the unweighted mean and Lat. the latency (ms) over the three single-image benchmarks.

Case Study

In Figure 6, we answer one PMC-VQA case with both interfaces, the backbone above and the deployed head below. Qwen3.5-4B spends 408.1 ms decoding ten tokens and names the bones of the forearm for an image of an ankle, while MedJev-4B scores the same candidates in 63.1 ms and puts 0.75 on the reference. The candidate the sentence names is the one the head ranks last at 0.06, and the largest share of what remains goes to the other pair of leg bones at 0.11. A sentence commits to one of these candidates without saying how close the others were, and that margin is exactly what a single forward pass returns for free. The decision is also 6.47 times faster than the sentence here, close to the 6.04 times that the four benchmarks average.

Figure 6. One PMC-VQA case under both interfaces. The backbone decodes a sentence that names the wrong bones, while MedJev-4B scores all four candidates in one pass and picks the reference.

Conclusion

In this paper, we presented MedJev, a system one vision-language decision model for fast and effective medical image understanding. Built on Qwen3.5-4B, MedJev-4B further employs language-side depth reduction and a residual decision head for low latency and accurate candidate scoring. Across extensive experiments on four medical benchmarks, the effectiveness and the efficiency of MedJev-4B are fully validated. In the future, we plan to extend our work to more general scenarios such as general vision-language decision models and complex visual perception tasks.

Introduction to the Datasets

We evaluate on four medical image understanding benchmarks. They differ in imaging modality, in the number of images a case carries, and in whether the candidates are a fixed label set or free text, and the number of candidates per example ranges from two to seven.

HAM10000 [39]. Dermoscopic images of pigmented skin lesions, each labelled with one of seven diagnostic categories. The categories are melanoma, melanocytic nevus, basal cell carcinoma, actinic keratosis or intraepithelial carcinoma, benign keratosis, dermatofibroma and vascular lesion. We use the dermoscopic images as the training pool and the ISIC 2018 Task 3 split as the labelled test set, which holds 1,512 cases. The categories are imbalanced, with melanocytic nevus covering most cases.

PneumoniaMNIST [50]. Pediatric chest radiographs at 224×224224 \times224 from MedMNIST, to be classified as normal or pneumonia. We use the official split, which holds 4,708 training and 624 test images.

MedThinkVQA [52]. Clinical cases drawn from radiological case reports, each presenting 2 to 47 images with a question and five candidate diagnoses. The split holds 7,347 training and 720 test cases, and the test cases carry 5,845 images in total, 8.12 per case on average.

PMC-VQA [55]. Questions about figures from the biomedical literature, each with four candidate answers written as free text and not drawn from a label set. We use version 2 of the dataset, whose test split holds 33,430 question-answer pairs over 29,021 figures.

Preprocessing. An example is kept when every image of it decodes, when it offers between two and seven candidates, and when the reference answer matches exactly one candidate. After these filters the four benchmarks contribute 174,360 training and 36,286 test examples, as listed in Table A1.

SettingValue
Retained Language Layers JJ23
Layer Supplying the Matching Projections32
Removed Complete Layers24–31
Residual Head Hidden Width256
Residual Head ActivationGELU
Residual Output InitializationZero
Maximum Candidates per Example7
LoRA Rank, Scaling, Dropout16, 32, 0
Visual Tokens per Image, per Example1,024, 4,096
Maximum Input Tokens32,768
Epochs, Global Batch Size, Steps1, 32, 5,449
Learning Rate, Weight Decay, Gradient Clip0.0001, 0.01, 1
Seed42
Training Examples174,360
HAM100009,951
PneumoniaMNIST4,698
MedThinkVQA7,307
PMC-VQA152,404
Test Examples36,286

Table A1. Training and model configuration of the deployed MedJev-4B, read from the frozen protocol of the training run that produced the deployed checkpoint of the paper.

Introduction to the Baselines

We compare against 20 off-the-shelf vision-language models at the 4B to 9B scale, and separately against six released decision models. Every baseline runs with its official weights and its own chat template, and each is timed in its own session under the protocol of Sec. C.

General Vision-Language Models. Fifteen models come from five families. From LLaVA we take LLaVA-1.5-7B, LLaVA-1.6-Vicuna-7B, LLaVA-1.6-Mistral-7B and LLaVA-OneVision-7B [19, 25, 26]. From InternVL we take InternVL2.5-4B, InternVL3-9B, InternVL3.5-4B and InternVL3.5-8B [7, 43, 58]. From Keye-VL we take Keye-VL-8B-Preview and Keye-VL-1.5-8B [16, 49], and from MiMo-VL we take MiMo-VL-7B-RL [46]. From Qwen we take Qwen2.5-VL-7B, Qwen3-VL-4B, Qwen3-VL-8B and Qwen3.5-9B [3, 4, 32], four models that span two generations.

Medical Vision-Language Models. Four models are trained on biomedical image-text data. They are LLaVA-Med-v1.5-7B [20], HuatuoGPT-Vision-7B [5], MedGemma-4B [34] and Lingshu-7B [48].

The Backbone. Qwen3.5-4B [31] is the model MedJEV-4B is built from. It shares the vision encoder, the connector and the retained language layers, and it is timed in the same session as MedJEV-4B.

Released Decision Models. Six systems share the decision interface. Two of them are medical, OmniMed-Jev-4B [37] and PathJev-P [38], both on a MedGemma-1.5-4B backbone. Four are general-purpose, Visual JEV-4B [53] on Qwen3-VL-4B, d1-3B [21] on LFM2.5-VL-3B, and Clef-Flash-9B [8] and d3-flash-9B [42] on Qwen3.5-9B. Each runs through the entry point its own release documents, at its own resolution and precision, and receives the candidate texts the benchmark defines.

Answer Extraction. A generative baseline returns a string, which is mapped to a candidate by one rule fixed before any baseline was run and applied unchanged to all of them. A response is accepted when it matches the text of exactly one candidate, when a line of it reads as an answer line, when its first or last line is a bare option letter, or when it carries an <answer> tag holding a letter. A response that names no candidate, names two, or whose letter and text disagree counts as an error.

Further Implementation Details

Table A1 lists the configuration of the deployed MedJev-4B, read from the frozen protocol.

Prompt and Markers. A request is laid out as the visual tokens of every image, the question, a question marker, then each candidate followed by an option marker, and finally a decision marker. The three markers are added to the vocabulary as <|medjev_question_end|>, <|medjev_option_end|> and <|medjev_decide|>, and their embeddings are initialized from the end-of-turn token of the backbone so that the first forward pass starts from a state the pretrained model already understands. Every image keeps all of the tokens the native vision merger produces, up to 1,024 tokens per image and 4,096 per example, and no token reduction or pooling is applied anywhere.

Depth Reduction. The backbone of Qwen3.5-4B [31] has 32 language layers. We keep layers 1 to 23 complete, delete layers 24 to 31 entirely, and keep from layer 32 only the query and key projections that the matching term needs. The last layer of Qwen3.5 is a full-attention layer and supplies the matching space. Its query projection also emits an output gate, which we discard.

Training. A single model is trained for one epoch over the 174,360 training examples of the four benchmarks, which is 5,449 updates at a global batch size of 32. The loss is a cross-entropy over the candidates of each example, weighted so that no benchmark dominates and, within each classification benchmark, so that no diagnosis dominates. Each benchmark first receives the same total weight, which scales an example by the pooled training size over four times its own, from 4.38 on HAM10000 to 0.29 on PMC-VQA. Within a benchmark, a classification example is weighted by the inverse of its observed class frequency and question-answering examples count equally. The LoRA [13] adapters are merged into the backbone before any timing.

Timing. Latency is measured on one H20 at batch size one, with a warmup that is excluded and with the GPU otherwise idle. The timed region starts when the request arrives and ends when the decision is on the CPU, so it includes reading and decoding the image, the processor, the host-to-device copy, the forward pass, and the final synchronisation. For the generative baselines the same region also contains the decoding loop and the parser.

Additional Results

Released Decision Models

Removing generation is necessary but not sufficient, and Table A2 is the evidence. Every model in it already returns scores instead of text, yet none is competitive here, which places our gain in how the decision is built and not in the interface alone. The per-benchmark view makes the failure mode explicit. d1-3B lands on exactly 37.50% on PneumoniaMNIST, the share of the smaller class, and on 59.92% on HAM10000, within three cases of the share of the largest class, so both numbers are what a constant answer would score. A released model can therefore post a plausible average while deciding almost nothing, and the margin that matters is the one over the majority-class floor, where MedJev-4B leads by 8.27% on HAM10000 and 31.73% on PneumoniaMNIST.

ModelBackboneAccuracy (%) HAMAccuracy (%) Pneu.Accuracy (%) MedThinkAccuracy (%) PMCLatency (ms) HAMLatency (ms) Pneu.Latency (ms) MedThinkLatency (ms) PMC
d1-3B [21]LFM2.5-VL-3B59.9237.5031.5355.4343.828.367.034.7
Visual Jev-4B [53]Qwen3-VL-4B56.8162.0220.5651.9988.569.5106.877.5
OmniMed-Jev-4B [37]MedGemma-1.5-4B66.6795.0322.2232.93204.0202.8207.1197.4
PathJev-P [38]MedGemma-1.5-4B51.3264.1026.8137.23194.0195.3196.8186.9
Clef-Flash-9B [8]Qwen3.5-9B61.9079.9728.0656.37112.161.4124.675.9
d3-flash-9B [42]Qwen3.5-9B67.0071.4732.9259.8899.860.3118.567.7
MedJev-4B (Ours)Qwen3.5-4B68.3994.2351.8161.6664.350.9260.655.1

Table A2. Released decision models, per benchmark. Accuracy and mean end-to-end latency, which the main paper summarises as one accuracy column and one latency column. Every row was measured in one session on one H20 at batch size one, through the entry point the release documents. The MedThinkVQA columns are scored from the first image of each case, so the input stays the same.

The multi-image column separates the systems further. Every row is scored from the first image of each case on MedThinkVQA [52], which keeps the input identical across interfaces and leaves 5,125 images unseen, and MedJev-4B still reaches 51.81% where the best released model reaches 32.92%. The deployed model reads every image of a case in one pass, which is why the design treats the number of images as a property of the request and not as a limit of the interface, and why the latency is averaged over the three single-image benchmarks and not over all four.

Confidence Calibration

Scoring every candidate gives a confidence signal for free, and Figure A1 (a) shows that the signal is usable and not decorative. Accuracy rises with confidence across the populated bins apart from one bin of five examples, and the expected calibration error over all 36,286 test examples is 6.18%, so the ordering the head produces indicates whether its decision should be trusted.

Figure A1. Calibration and per-category behaviour of the deployed model. (a) Accuracy against confidence in fifteen equal-width bins over all 36,286 test examples. (b) Accuracy on the answered cases when the least confident ones are referred away. (c) The confusion on HAM10000, each row normalized by the test cases of that category. (d) The recall of each category with its counts.

What this buys a deployment is in Figure A1 (b). Answering only the cases above a confidence threshold raises accuracy from 62.33% at full coverage to 92.67% on the most confident 14.4% of cases, so the same model can be operated at a chosen reliability by referring the rest to a clinician. A model that answers in free text gives one answer and no ranking over the alternatives, so this control costs a pass.

Candidate Order

Table A3 | Candidate order, in full. Every test example is decided again with the candidate order permuted, under three seeds. Kept, Lost and Gained count the natively correct decisions that survive, those that break, and the natively wrong decisions that become correct, pooled over the three seeds.

A model that scores candidates could be reading the slot a candidate occupies instead of the text in it, and Table A3 bounds how much of that is happening. Decided again under three permuted orders, the unweighted mean falls from 69.06% to 68.10%, the largest single drop is 2.03% on MedThinkVQA, and PneumoniaMNIST does not move at all, so position accounts for at most a small part of it.

BenchmarknnNativePermuted (3 Seeds)SpreadKeptLostGained
HAM10000 [39]1,51268.3968.19 / 67.92 / 67.720.463,0544828
PneumoniaMNIST [50]62494.2394.23 / 94.23 / 94.230.001,76400
MedThinkVQA [52]72051.9449.86 / 50.00 / 49.860.14975147103
PMC-VQA [55]33,43061.6860.19 / 60.48 / 60.250.2856,0155,8454,464

Table A3. Candidate order, in full. Every test example is decided again with the candidate order permuted, under three seeds. Kept, Lost and Gained count the natively correct decisions that survive, those that break, and the natively wrong decisions that become correct, pooled over the three seeds.

The churn underneath that stability is the more informative part. On PMC-VQA the permutation costs 5,845 correct decisions and returns 4,464 that were wrong before, a net loss of 1,381 out of 100,290 draws. The two directions are close in size, so the order rearranges which examples are decided correctly far more than it changes how many of them there are.

Per-Category Behaviour on HAM10000

Per-Category Behaviour on HAM10000

An average over seven categories can hide where a model fails, so Figure A1 (c) and (d) open it up, and the accuracy turns out to be concentrated where the clinical cost is highest. MedJev-4B recalls melanoma in 146 of 171 cases and basal cell carcinoma in 71 of 93, while still recalling the dominant benign category of melanocytic nevus in 628 cases out of 909.

The errors also fall in the direction a screening setting prefers. The confusion is asymmetric towards the malignant categories, with 25% of the nevus cases sent to melanoma and far fewer in the opposite direction, so the errors that remain fall mostly on the side of over-referral. That asymmetry belongs to the decision and not to the interface, since every category is in the candidate set and always scored.

Per-Request Latency

A mean latency hides the behaviour a deployment has to budget for, and Table A4 shows that the two interfaces differ in kind. For the generative models the 99th percentile runs far above the median, because the cost is set by the response length, reaching tens of seconds on MedThinkVQA.

ModelHAM10000PneumoniaMNISTMedThinkVQAPMC-VQA
Med.p99Tok.Med.p99Tok.Med.p99Tok.Med.p99Tok.
MedGemma-4B [34]21422332062133809331732132233
Qwen3.5-4B [31]208457426031163542781443617908
MedJev-4B (Ours)7284062780207625063810

Table A4. Table A4 | Per-request latency and generated tokens. Median and 99th percentile of the end-to-end latency in milliseconds, and the median number of generated tokens, over 150 timed requests per benchmark and model on one H20 at batch size one. Warmups are excluded, thinking is off, and the three requests MedGemma-4B failed on PMC-VQA are left out of its own statistics.

MedJev-4B has almost no tail, and its median and its 99th percentile sit within twenty milliseconds of each other on the three single-image benchmarks, while the multi-image MedThinkVQA cases stay inside one second at the 99th percentile. The work it does is fixed by the request and not by the answer it is about to give, which makes the cost of a decision predictable in advance.

Additional Visualisations

The main paper reduces all four benchmarks together, which is the right view for its claim that training moves the four sources into one shared region. Figure A2 (a) and Figure A2 (b) reduce each benchmark on its own with t-SNE [41] instead, so that a panel cannot be arranged by the other three. The shift from the pretrained states to the trained ones holds inside every benchmark, and the split between correct and incorrect candidates is visible in each of them, which rules out the reading that the separation in the pooled figure is an artefact of the benchmarks sitting in different regions.

Figure A2. Hidden representations in detail. Each panel is reduced on its own, so its layout is never driven by the others. (a) and (b) give one column per benchmark, the decision marker state at layer 23 before and after training, and the candidate marker states of the trained model split by whether the candidate is correct. (c) gives the same split at four depths of the pretrained backbone.

Figure A2 (c) looks at the pretrained backbone alone, with none of our training applied. The correct candidates are already pulled away from the incorrect ones at depths well below the top of the stack, and the geometry does not sharpen much over the last layers. This is the same observation the depth probe of the main paper makes with a linear readout, seen here without fitting anything, and it is the reason why a model that keeps only 23 layers loses no accuracy.

References

  1. [1]Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyansky, Y., Lebrón, F., & Shanghai, S. (2023). GQA: Training generalized multi-query transformer models from multi-head.
  2. [2]Almeida, D. (2026). Introducing System One models & Jev. TypeSafe AI Blog. https://typesafe.ai/blog/introducing-system-one-models-and-jevtypesafe.ai/blog/introducing-system-one-models-and-jev
  3. [3]Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., & al., . et . (2025). Qwen3-VL technical report. In arXiv preprint. https://arxiv.org/abs/2511.21631
  4. [4]Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., & al., . et . (2025). Qwen2.5-VL technical report. In arXiv preprint. https://arxiv.org/abs/2502.13923
  5. [5]Junying Chen, Chi Gui, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xidong Wang, Ruifei Zhang, Zhenyang Cai, Ke Ji, Guangjun Yu, Xiang Wan, and Benyou Wang. HuatuoGPT-Vision, towards injecting medical visual knowledge into multimodal LLMs at scale. *arXiv preprint arXiv:2406.19280*, 2024. URL https://arxiv.org/abs/2406.19280.
  6. [6]Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In *European Conference on Computer Vision (ECCV)*, 2024. URL https://arxiv.org/abs/2403.06764.
  7. [7]Chen, Z., Wang, W., Cao, Y., Liu, Y., & al., . et . (2024). Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. In arXiv preprint. https://arxiv.org/abs/2412.05271
  8. [8]Cloudflare. (2026). Introducing Clef: Our open-source decision models, and new RL fine-tuning platform. Blog Post. https://blog.cloudflare.com/clef-decision-models/blog.cloudflare.com/clef-decision-models
  9. [9]Jonathan St. B. T. Evans and Keith E. Stanovich. Dual-process theories of higher cognition: Advancing the debate. *Perspectives on Psychological Science*, 8(3):223–241, 2013. doi: 10.1177/1745691612460685. URL https://doi.org/10.1177/1745691612460685.
  10. [10]Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A. Roberts. The unreasonable ineffectiveness of the deeper layers. In *International Conference on Learning Representations (ICLR)*, 2025. URL https://arxiv.org/abs/2403.17887.
  11. [11]Guo, D., Yang, D., Zhang, H., & al., . et . (2025). DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 645(8081), 633–638. https://doi.org/10.1038/s41586-025-09422-z
  12. [12]Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. PathVQA: 30000+ questions for medical visual question answering. *arXiv preprint arXiv:2003.10286*, 2020. URL https://arxiv.org/abs/2003.10286.
  13. [13]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In *International Conference on Learning Representations (ICLR)*, 2022. URL https://arxiv.org/abs/2106.09685.
  14. [14]Yutao Hu, Tianbin Li, Quanfeng Lu, Wenqi Shao, Junjun He, Yu Qiao, and Ping Luo. OmniMed-VQA: A new large-scale comprehensive evaluation benchmark for medical LVLM. arXiv preprint arXiv:2402.09181, 2024. URL https://arxiv.org/abs/2402.09181.
  15. [15]Daniel Kahneman. Thinking, Fast and Slow. Farrar, Straus and Giroux, New York, 2011.
  16. [16]Kwai Keye Team. Kwai Keye-VL technical report. arXiv preprint arXiv:2507.01949, 2025. URL https://arxiv.org/abs/2507.01949.
  17. [17]Yuxiang Lai, Jike Zhong, Ming Li, Shitian Zhao, Yuheng Li, Konstantinos Psounis, and Xiaofeng Yang. Med-R1: Reinforcement learning for generalizable medical reasoning in vision-language models. arXiv preprint arXiv:2503.13939, 2025. URL https://arxiv.org/abs/2503.13939.
  18. [18]Jason J. Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images. Scientific Data, 5: 180251, 2018. doi: 10.1038/sdata.2018.251.DOI
  19. [19]Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z., & al., . et . (2024). LLaVA-OneVision: Easy visual task transfer. In arXiv preprint. https://arxiv.org/abs/2408.03326
  20. [20]Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. LLaVA-Med: Training a large language-and-vision assistant for biomedicine in one day. arXiv preprint arXiv:2306.00890, 2023. URL https://arxiv.org/abs/2306.00890.
  21. [21]AI, L. (2026). Open d1: Edge decision models for text, vision, and audio. Blog Post. https://www.liquid.ai/blog/d1-openliquid.ai/blog/d1-open
  22. [22]Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. SLAKE: A semantically-labeled knowledge-enhanced dataset for medical visual question answering. In IEEE International Symposium on Biomedical Imaging (ISBI), 2021. URL https://arxiv.org/abs/2102.09542.
  23. [23]Bo Liu, Lianyu Wang, Meng Wang, Ching-Yu Cheng, and Huazhu Fu. RetiJev: Fast and source-grounded ophthalmic decision models. arXiv preprint arXiv:2610.30947, 2026. URL https://arxiv.org/abs/2610.30947.
  24. [24]Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems (NeurIPS), 2023. URL https://arxiv.org/abs/2304.08485.
  25. [25]Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. URL https://arxiv.org/abs/2310.03744.
  26. [26]Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., & Lee, Y. J. (2024). LLaVA-NeXT: Improved reasoning, OCR, and world knowledge. Blog Post. https://llava-vl.github.io/blog/2024-01-30-llava-next/llava-vl.github.io/blog/2024-01-30-llava-next
  27. [27]Men, X., Xu, M., Zhang, Q., Wang, B., Lin, H., Lu, Y., Han, X., & Chen, W. (2024). ShortGPT: Layers in large language models are more redundant than you expect. In arXiv preprint. https://arxiv.org/abs/2403.03853
  28. [28]Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Cyril Zakka, Yash Dalmia, Eduardo Pontes Reis, Pranav Rajpurkar, and Jure Leskovec. Med-Flamingo: A multimodal medical few-shot learner. arXiv preprint arXiv:2307.15189, 2023. URL https://arxiv.org/abs/2307.15189.
  29. [29]Jiazhen Pan, Che Liu, Junde Wu, Fenglin Liu, Jiayuan Zhu, Hongwei Bran Li, Chen Chen, Cheng Ouyang, and Daniel Rueckert. MedVLM-R1: Incentivizing medical reasoning capability of vision-language models (vlms) via reinforcement learning. arXiv preprint arXiv:2502.19634, 2025. URL https://arxiv.org/abs/2502.19634.
  30. [30]Luozheng Qin, Yuhan Zhu, Changhai Zhou, Qian Qiao, Jun Gao, Yuhua Zhou, Harry Zhang, Xiangyu Zeng, Yuxuan Liu, Yuxin Lu, Kaijie Chen, Andrew Chen, and Pony Ma. MJev-Spatial: System One spatial intelligence via native attention matching. arXiv preprint arXiv:2610.22276, 2026. URL https://arxiv.org/abs/2610.22276.
  31. [31]Team, Q. (2026). Qwen3.5-4B: Official model card. Hugging Face. https://huggingface.co/Qwen/Qwen3.5-4Bhuggingface.co/Qwen/Qwen3.5-4B
  32. [32]Team, Q. (2026). Qwen3.5-9B: Official model card. Hugging Face. https://huggingface.co/Qwen/Qwen3.5-9Bhuggingface.co/Qwen/Qwen3.5-9B
  33. [33]Joshua Robinson, Christopher Michael Rytting, and David Wingate. Leveraging large language models for multiple choice question answering. In International Conference on Learning Representations (ICLR), 2023. URL https://arxiv.org/abs/2210.12353.
  34. [34]Sellergren, A., Kazemzadeh, S., Jaroensri, T., Kiraly, A., Traverse, M., Kohlberger, T., Xu, S., Jamil, F., Hughes, C., Lau, C., & al., . et . (2025). MedGemma technical report. In arXiv preprint. https://arxiv.org/abs/2507.05201
  35. [35]Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. LLaVA-PruMerge: Adaptive token reduction for efficient large multimodal models. In IEEE/CVF International Conference on Computer Vision (ICCV), 2025. URL https://arxiv.org/abs/2403.15388.
  36. [36]Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. RoFormer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864, 2021. URL https://arxiv.org/abs/2104.09864.
  37. [37]Luyao Tang and Cheng Chen. OmniMed-Jev: Calibrating LVLM confidence for trustworthy medical multimodal decisions via system one. arXiv preprint arXiv:2610.00381, 2026. URL https://arxiv.org/abs/2610.00381.
  38. [38]Cristian Tommasino and Antonio M. Rinaldi. PathJev: Decision-native vision-language modelling for histopathology. arXiv preprint arXiv:2610.32478, 2026. URL https://arxiv.org/abs/2610.32478.
  39. [39]Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific Data, 5: 180161, 2018. doi: 10.1038/sdata.2018.161. URL https://www.nature.com/articles/sdata2018161.DOI
  40. [40]Tu, T., Azizi, S., Driess, D., Schaekermann, M., Amin, M., Chang, P.-C., Carroll, A., Lau, C., Tanno, R., Ktena, I., & al., . et . (2023). Towards generalist biomedical AI. In arXiv preprint. https://arxiv.org/abs/2307.14334
  41. [41]Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. *Journal of Machine Learning Research*, 9(86):2579–2605, 2008. URL https://jmlr.org/papers/v9/vandermaaten08a.html.jmlr.org/papers/v9/vandermaaten08a.html
  42. [42]Team, . vLLM S. R. (2026). d3-flash: A multimodal foundation decision model. In Model release. https://huggingface.co/vllm-sr/d3-flashhuggingface.co/vllm-sr/d3-flash
  43. [43]Wang, W., Gao, Z., Gu, L., Pu, H., Cui, L., Wei, X., Liu, Z., Jing, L., Ye, S., Shao, J., & al., . et . (2025). InternVL3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. In arXiv preprint. https://arxiv.org/abs/2508.18265
  44. [44]Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In *Advances in Neural Information Processing Systems (NeurIPS)*, 2022. URL https://arxiv.org/abs/2201.11903.
  45. [45]Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Towards generalist foundation model for radiology by leveraging web-scale 2D&3D medical data. *arXiv preprint arXiv:2308.02463*, 2023. URL https://arxiv.org/abs/2308.02463.
  46. [46]Team, X. L.-C. (2025). MiMo-VL technical report. In arXiv preprint. https://arxiv.org/abs/2506.03569
  47. [47]Xing, L., Huang, Q., Dong, X., Lu, J., Zhang, P., Zang, Y., Cao, Y., He, C., Wang, J., Wu, F., & al., . et . (2025). Conical visual concentration for efficient large vision-language models. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). https://arxiv.org/abs/2410.17247
  48. [48]Weiwen Xu, Hou Pong Chan, Long Li, Mahani Aljunied, Ruifeng Yuan, Jianyu Wang, Chenghao Xiao, Guizhen Chen, Chaoqun Liu, Zhaodonghui Li, Yu Sun, Junao Shen, Chaojun Wang, Jie Tan, Deli Zhao, Tingyang Xu, Hao Zhang, and Yu Rong. Lingshu: A generalist foundation model for unified multimodal medical understanding and reasoning. *arXiv preprint arXiv:2506.07044*, 2025. URL https://arxiv.org/abs/2506.07044.
  49. [49]Yang, B., Wen, B., Ding, B., Liu, C., & al., . et . (2025). Kwai Keye-VL 1.5 technical report. In arXiv preprint. https://arxiv.org/abs/2509.01563
  50. [50]Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. MedMNIST v2—a large-scale lightweight benchmark for 2D and 3D biomedical image classification. *Scientific Data*, 2023. doi: 10.1038/s41597-022-01721-8. URL https://arxiv.org/abs/2110.14795.
  51. [51]Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. VisionZip: Longer is better but not necessary in vision language models. *arXiv preprint arXiv:2412.04467*, 2024. URL https://arxiv.org/abs/2412.04467.
  52. [52]Zonghai Yao, Benlu Wang, Yifan Zhang, Junda Wang, Iris Xia, Zhipeng Tang, Shuo Han, Feiyun Ouyang, Zhichao Yang, Arman Cohan, and Hong Yu. Medical thinking with multiple images. In *International Conference on Learning Representations (ICLR)*, 2026. URL https://arxiv.org/abs/2604.16506.
  53. [53]Guanxu Yu and Yuhang Yao. Visual Jev: Accurate and efficient decisions from shared visual context. *arXiv preprint arXiv:2609.25845*, 2026. URL https://arxiv.org/abs/2609.25845.
  54. [54]Zhang, S., Xu, Y., Usuyama, N., Xu, H., Bagga, J., Tinn, R., Preston, S., Rao, R., Wei, M., Valluri, N., & al., . et . (2023). BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. In arXiv preprint. https://arxiv.org/abs/2303.00915
  55. [55]Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. PMC-VQA: Visual instruction tuning for medical visual question answering. arXiv preprint arXiv:2305.10415, 2023. URL https://arxiv.org/abs/2305.10415.
  56. [56]Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, and Shanghang Zhang. Sparse-VLM: Visual token sparsification for efficient vision-language model inference. In International Conference on Machine Learning (ICML), 2025. URL https://arxiv.org/abs/2410.04417.
  57. [57]Xunlan Zhou, Xianliang Yang, and Li Zhao. From text decisions to pixels: An study of Jev-style visual choice model. arXiv preprint arXiv:2609.29283, 2026. URL https://arxiv.org/abs/2609.29283.
  58. [58]Zhu, J., Wang, W., Chen, Z., Liu, Z., & al., . et . (2025). InternVL3: Exploring advanced training and test-time recipes for open-source multimodal models. In arXiv preprint. https://arxiv.org/abs/2504.10479
  59. [59]Yuhan Zhu, Xiangyu Zeng, Qingyi Si, Chenxu Yang, Yifei Li, Luozheng Qin, Junhao Zhou, Dingyu Yao, Jun Zhang, Chuanyu Qin, Zhiqiu Zhang, Yuandong Yang, Yuqi Xu, Tianyu Zhang, Hongcheng Guo, Jiaqi Wang, Nan Duan, and Limin Wang. VideoJev: System One model for video understanding and interaction. arXiv preprint arXiv:2610.18615, 2026. URL https://arxiv.org/abs/2610.18615.

Paper details

vlnc:W_4gwe3g99
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents