nnInteractive Agent: General-Purpose AI Agents as 3D Image Annotators

Introduction

Segmentation turns images into quantitative measurements such as organ volumes, tumour burden or cell counts. Dedicated models segment the structures they were trained on; everything else is delineated by people, increasingly with the help of promptable models. nnInteractive [4], for example, predicts a full 3D mask from points, scribbles, boxes or lassos placed in single slices and was trained on more than 120 volumetric datasets spanning CT, MRI, PET and 3D microscopy. Such a model makes every prompt count, but the annotation process itself remains with a human: deciding what the target is and where it lies, placing prompts, judging the 3D result, correcting it, and revising it when someone asks for changes.

Text-prompted medical segmentation models take the human out of this loop [5, 6, 7, 8]. They map the name or description of a structure directly to a mask and can be very accurate, but their accuracy depends on how close a request is to their training data.

General-purpose AI agents such as Claude Code and Codex combine a multimodal large language model with tool use and code execution. They interpret images, have broad anatomical knowledge, follow multi-step instructions, write code and can check their own work. They cannot, however, produce a voxel-accurate 3D mask by themselves. A promptable model like nnInteractive can, but it cannot decide what to segment. The two are complementary.

Combining them, we let an agent take over the annotation process. The user makes a request in plain language; the agent works out what has to be segmented, finds the targets, prompts nnInteractive, inspects the 3D result, corrects it and responds to feedback, while nnInteractive turns each prompt into a 3D mask (Figure 1). Asked for the muscles used for chewing, the agent works out which muscles these are, segments them and explains why it left one out; asked to count liver lesions, it writes code to find candidates, flags a lesion that may be two, and splits it when the user asks (Figure 5). No component is trained for this purpose. nninteractive-agent provides nnInteractive as a set of tools over the Model Context Protocol (MCP) [9], together with a skill, a document that describes how experienced users prompt nnInteractive.

We evaluate nninteractive-agent with Claude Code and Codex on a benchmark of 14 images and compare the agents with simulated prompts and with a text-prompted model. Agents that call segmentation models exist, including training-free agents for 3D images (Section 2). What we add is an extension that lets an unmodified general-purpose agent carry out the interactive 3D annotation process itself (finding, prompting, checking, correcting) across modalities, species and structures, and an evaluation of its segmentation accuracy. The tool, the skill, a web demo and the benchmark are open source.

Related work

Promptable segmentation. SAM [10] established segmentation from points and boxes in 2D. There have also been some notable evaluations and adaptations in the medical domain [11, 12, 13]. In 3D, SAM-Med3D [14] and VISTA3D [15] accept point prompts, SegVol [5] points and boxes, and MedSAM2 [16] propagates a box drawn on one slice through the volume. nnInteractive [4] accepts points, scribbles, boxes and lassos placed in single slices and predicts a full 3D mask. In their interactive modes these models are class-agnostic: what is segmented is decided by whoever places the prompts, usually a human. This makes them a natural tool for an agent. We use nnInteractive for three reasons: it supports the widest range of prompt types among these models, including scribbles, which describe elongated or irregular structures that a single point or box cannot; it clearly outperformed existing promptable 3D methods in extensive benchmarking [4]; and it is open-set, segmenting whatever the prompts describe, and was trained across CT, MRI, PET and 3D microscopy.

Text-prompted segmentation. SegVol [5], SAT [6] and VoxTell [8] map the name or description of a structure directly to a 3D mask, and SAM 3 [17] does so for concepts in natural images and videos. These models need no operator, and some are designed to generalise beyond their training data: VoxTell, our text-prompted baseline, reports strong zero-shot results on unseen datasets and on classes related to those it was trained on. Their accuracy on medical images still depends on how close a request is to the training distribution, whereas that of an agent operating a class-agnostic model depends on whether it recognises the target in the image and prompts the model well.

Agents that prompt segmentation models. Closest to our work are agents that operate a promptable model. SegAgent [18], MedSAM-Agent [19] and IBISAgent [20] train multimodal models on annotation trajectories, partly with reinforcement learning, to issue clicks and boxes to SAM-type models on 2D images. Our agent, in contrast, is not trained at all: it learns how to prompt from the skill and keeps its general abilities. The SAM 3 Agent [17] and the MedSAM-3 Agent [21] use off-the-shelf multimodal models, but they prompt a concept segmenter with text and inspect the returned masks on 2D images, leaving the spatial decisions to the segmenter. Seg-Agent [22], also training-free, does make the spatial decisions: it places visual prompts for SAM and refines them after inspecting candidate masks drawn on 2D natural images. In MedOpenClaw [23], frontier agents operate 3D Slicer and can call VISTA3D, but segmentation is an auxiliary step for diagnostic questions, its accuracy is not evaluated, and the authors found the agents’ multi-step segmentation workflows unreliable. In our work, interactive 3D segmentation is the task itself: the agent places spatial prompts, including scribbles, checks the result in rendered slices and 3D views, and corrects it. We evaluate this process for segmentation accuracy across imaging modalities and species.

Training-free agents and imaging software. Agents built on frontier models without additional training are increasingly used in medical imaging, also for 3D images. Endur et al. [24] run brain MRI pipelines (registration, automatic tumour and anatomy segmentation, volumetry) through text-only tool calls, and CT-Flow [25] provides CT viewing, measurement and automatic segmentation tools through MCP servers for question answering and reporting. HealthAgentBench [26] evaluates unmodified Claude Code and Codex on healthcare tasks and finds imaging tasks, including 3D CT, the hardest, which suggests that agents need suitable tools for images. Agents can also operate imaging software directly, by writing code in napari [27] or through generic MCP servers for 3D Slicer and napari [28, 29]. Most of these systems rely on automatic segmentation models, which limits them to the structures those models know, and many are dedicated agent frameworks around their own tools or viewers. nnInteractive-agent instead extends an existing general-purpose agent: an MCP server and a skill that any Claude Code or Codex session can load, without a viewer.

nninteractive-agent

Division of labour

The agent decides what to segment and where to prompt, and it judges the result; nnInteractive turns each prompt into a 3D mask (Fig. 2). The agent never draws a mask itself and never needs to be precise to the voxel: a scribble along the middle of a structure is enough for nnInteractive to find its boundary in 3D. Conversely, nnInteractive needs no knowledge of what a “left parotid gland” is: the agent supplies it through the location of its prompts. This split puts each capability where it is strongest, and nothing in the system is specific to an anatomy, a modality or a species.

Tools and workspace

The MCP server keeps one workspace per agent session: the image, one nnInteractive session, the object being segmented with its prompts, and an instance map of the objects committed so far. The agent works with 19 tools in four groups.

Objects. nnInteractive segments one object at a time. The agent starts an object, adds prompts, and commits it to the instance map, where every voxel belongs to one object. A structure inside another one, such as a lesion in an organ, takes those voxels from the outer object when the agent asks for it; otherwise an overlap with a committed neighbour is treated as a leak and reported. Committed objects can be reopened for corrections.

Prompts. A point marks a tiny object or a tiny error; a box encloses the object in one slice and is allowed only as an object’s first prompt; a scribble is a line drawn in one slice. nnInteractive accepts any binary mask as a scribble; the agent specifies a polyline, which the tool rasterises 2 pixels wide. Points and scribbles can be positive or negative. Boxes and an object’s first scribble are previewed: the tool draws them on a zoomed view without sending them, so the agent can check the coordinates it read off a rendered image. After every prompt the tool reports how the mask changed and flags common problems, such as a prediction that extends past the box it came from.

Views. The agent sees the image only through rendered views (Figure 3; more examples in Figure S6–Figure S9 of the supplement): orthogonal slices with absolute voxel coordinates on the axes, patient directions and object outlines; montages of many slices; projections; and 3D surface renderings of an object from several directions. Statistics (size, connected components, overlaps) complement them.

Python. A persistent Python namespace gives the agent the image array and the workspace. The agent uses it to find candidates (thresholds, connected components, maximum intensity projections), to loop over many instances, and to draw its own figures (Figure 5a). The tool can be restricted to computations or switched off.

• One object at a time. Each object is a single structure instance. Segmenting "the rib cage" means one object per rib [...]

• 2D prompts, 3D result. A prompt lives in one slice, but the prediction is a full 3D mask. Always check slices other than the one you prompted in.

• Prompts accumulate. [...] undo it right away if a prompt made things worse.

• Prompt types: scribbles preferred, boxes only where they fit, points rarely.

• Long lines work best: run the line along the middle of the region it describes and let it cover most of the region's visible extent [...]

• You decide what belongs to the object. [...] state the decision in your report.

• Look for missing parts first. A result that looks clean is often too small [...]

• Correct the errors from largest to smallest, one prompt at a time, re-checking after each [...]

• To check more than about 4 objects, split them into batches of 3–6. Verifier subagents can check the batches in parallel [...]

• [...] summarize what was segmented, how confident you are per object, and anything a human should review.

Objects larger than about 320 voxels along an axis are segmented on a downsampled copy of the image and scaled back up, since nnInteractive predicts $192^3$ patches and is most accurate on objects that fit into one patch or require only small AutoZoom factors. The agent only provides a size estimate and continues to work in full-resolution coordinates.

The skill

Tools alone do not make a good annotator. The skill, a Markdown document that agents load when asked to segment, encodes how experienced users work with nnInteractive. It describes how nnInteractive behaves, how to read coordinates off rendered views, a step-by-step workflow, a playbook for corrections and rules for working efficiently (Figure 4). Skills of this kind have also been shown to help agents operate scientific visualisation software [33].

Deployment

nninteractive-agent setup claude (or codex) registers the MCP server and installs the skill, so that every new session can segment images; any other MCP client can use the server, and clients without skill support read the skill as an MCP prompt. nnInteractive runs in the server’s process on a local GPU or on a shared nninteractive-server. Open models behind an Anthropic-compatible API can drive Claude Code as well. As a proof of concept, Qwen3.8-27B segmented a spleen (MSD Spleen, case 10 [30, 31]) with Dice 0.93 in 34 minutes, and the benchmark’s bladder tumour, with the location given, with Dice 0.60 (Claude Code and Codex: 0.72 to 0.81). It ran on a shared server under high load, so we did not pursue this further. A web demo with 2D and 3D views and a chat lets people use the agent with immediate visual feedback (Figure S5 in the supplement).

Agentic behaviour in practice

Before we measure how accurately agents segment named targets (§ 5), we show what an agent does with requests as people make them: often less specific, sometimes defined by function rather than by name, and with feedback along the way. The following examples come from the web demo with Claude Code (Opus 5.5, medium effort).

Finding and counting liver lesions

Asked only to “segment and count all liver lesions” in a contrast-enhanced CT (MSD Liver, case 155 [30, 31]), the agent made its own plan (Figure 5, box below). It first segmented the liver as a helper object. In Python, it smoothed the image, thresholded dark voxels inside the eroded liver mask, labelled the connected components and drew a figure of candidate regions for itself (Figure 5a). It then segmented each lesion as its own nnInteractive object, mostly starting from a box, separated two touching nodules with negative scribbles, reviewed the remaining candidates (dismissing vessels, fissures and noise), and deleted the helper. After three minutes and ten prompts it reported six definite lesions and one possible one, and flagged that lesion 6 might be two touching nodules. When told “lesion 6 seems to be a merger. Split into 2”, it checked the split with an intensity-based watershed of its own, reopened the object, removed the lateral nodule with a negative scribble, segmented that nodule as a new object, and confirmed that both boundaries agreed (Figure 5c,Figure 5d); this took 46 seconds and two prompts.

Conversation excerpt (shortened)

User: segment and count all liver lesions

Agent: I’ve got a liver mask; now finding hypodense lesion candidates inside it. [. . . ] I count 6 definite liver lesions and 1 possible one. [. . . ] To find them, I outlined the liver as a temporary helper (deleted afterwards), searched it for dark regions, and looked at each candidate in zoomed slices. [. . . ] Whether #6 is one lesion or two: at axis0 158–162 it shows as two touching nodules that merge higher up. I kept it as one mass.

User: lesion 6 seems to be a merger. Split into 2

Agent: Splitting #6 into a lateral and a medial nodule. These are clearly separate at axis0 158–162. [. . . ] The split agrees with a separate check based on image brightness, which put the boundary in the same place. Count: 7 definite liver lesions (#1–6, #8), plus the uncertain small spot #7.

Requests that need knowledge

Four further requests (Figure 1) need knowledge rather than search. The first three need anatomical knowledge; the fourth is not a medical image at all:

• “Segment the muscles used for chewing.” (TotalSegmentator MRI, s0506 [1]): the agent translated function into anatomy and segmented both masseters and both medial and lateral pterygoids (2 minutes 18 seconds, 7 prompts). It left out the temporalis and explained why: it “lies almost entirely above the scanned region”.

• “Now segment the muscles used for turning the head.”: both sternocleidomastoids (1 minute 30 seconds, 2 prompts). The agent named the deeper rotators it did not segment, because their borders were too faint on this scan, and offered to try.

• “Segment every rib as a separate object and name them by side and number.” (RibFrac, case 100 [2, 3]): the agent found 12 rib candidates per side with its own bone thresholding and prompted each with points along it. It noticed that its first pass had taken both collarbones for rib 1 and missed the right fifth rib under the shoulder blade, and its batch-wise reviews found that most ribs stopped short of the spine. It fixed all of this, rebuilt three ribs whose extensions had spilled into the spine, and reported that the numbering was counted from the first rib and not checked against the vertebrae. The result is 24 named ribs after 10.6 minutes and 223 prompts ($3.86 at API prices).

• “snorkel and flashlight” (a CT scan of a packed suitcase, image provided by Smiths Detection): an image far from the medical and microscopy images nnInteractive was trained on. The agent recognised the snorkel as the J-shaped tube along one side of the case and segmented it, including its mouthpiece, with four scribbles. For the flashlight it chose a dense metal cylinder after ruling out several other items, and said it was unsure: it saw no lens, reflector or switch, and the cylinder could also be a large battery (3 minutes 12 seconds, 5 prompts). Told that this was likely the battery and that the flashlight, missing its battery, was somewhere else, the agent found an empty tube with a flared head next to it, which it had taken for a cup-like container before. It segmented the tube with one scribble and re-created the first object as “battery” (46 seconds, 2 prompts).

Benchmark

Images and targets. The benchmark consists of 14 public images: six CT, five MRI, one PET and two micro-CT, of humans, a mouse and an ant (Figure 6, Table S1). Its 51 targets range from organs and tumours to muscles, bones, vessel trees and an insect brain. Every target is identifiable from its description alone: it is named, unique within the image, and its side is specified where relevant. Besides all 14 images, we report two groups: the nine images of human CT, MRI and PET of the head and neck, chest, abdomen and pelvis (radiology; 25 targets); the typical domain of medical segmentation models, and five images whose targets lie outside it (low muscle, ant; 26 targets): single muscles of the thigh and calf in fat-weighted research MRI, the 17 bones of the left foot in a whole-body CT of a cadaver, and a tumour and a brain in micro-CTs of a mouse and an ant. nnInteractive was not trained on any of these images, so the agents’ results are not inflated by training-data overlap; several sources are part of nnInteractive’s training data (§ 6.3). The references of the foot were made by the source’s annotators with nnInteractive in 3D Slicer and checked by them, which may favour the tracks that use nnInteractive.

Image	Targets	Source, licence
pediatric_ct	liver, spleen, right kidney, bladder, heart, pancreas, gallbladder, left adrenal	Pediatric-CT-SEG (TCIA) [34], CC BY 4.0
hcc_ct	liver, liver tumour	HCC-TACE-Seg (TCIA) [35], CC BY 4.0
adrenal_ct	adrenal tumour	Adrenal-ACC-Ki67-Seg (TCIA) [36], CC BY 4.0
lung_ct	lung tumour	RIDER Lung CT, version 3 (TCIA) [37], CC BY 4.0; tumour mask from RIDER-LungCT-Seg (TCIA) [38], CC BY 3.0
lung_pet	FDG-avid tumour	ACRIN-NSCLC-FDG-PET (TCIA) [39], CC BY 3.0; lesion mask from the BAMF/AIMI annotations [40, 41], CC BY 4.0
headneck_mr	mandible, brainstem, left parotid, right submandibular gland, thyroid, spinal cord, left optic nerve	HaN-Seg [32], CC BY-NC-ND 4.0
headneck_tumor_mr	primary tumour	HNTS-MRG 2024 [42], CC BY-NC 4.0
bladder_mr	bladder tumour	FedBCa [43], CC BY 4.0
thigh_mr	rectus femoris, sartorius, gracilis, vastus lateralis	Thigh and calf muscle MRI [44], on request from the authors (former figshare release: CC BY 4.0, non-commercial use only)
calf_mr	soleus, tibialis anterior, medial gastrocnemius	same
foot_ct	the 17 bones of the left foot: talus, calcaneus, navicular, cuboid, three cuneiforms, five metatarsals, the phalanges of each toe	Visible Human Full-Skeleton 3D Bone Models (Hugging Face) [45], CC BY 4.0, on the CT of the NLM Visible Human Female as aligned by [46], CC BY 4.0; courtesy of the U.S. National Library of Medicine
liver_vessels_ct	liver, portal vein, hepatic veins	VEELA 2025 [47, 48]; images from CHAOS [49], CC BY-NC-SA 4.0; no licence stated for the VEELA labels
mouse_microct	subcutaneous tumour	Subcutaneous tumour micro-CT database (ERDA) [50], CC BY (version not stated)
ant_microct	brain	Ant brains in micro-CT (Dryad) [51, 52], CC0 1.0

Tracks.

• Unguided: the agent gets the image and the list of targets by name (Figure S1 in the supplement). It has to find and segment them.

• Guided: the same, plus one voxel inside each target (the voxel deepest inside the reference). This separates segmenting from finding.

• One point: no agent; one positive point at the guided voxel per target. This is nnInteractive with the simplest possible prompt.

• Oracles: no agent; up to five simulated prompts per target with knowledge of the reference, each aimed at the largest error of the current mask (the largest connected region it misses or adds; before the first prompt, the whole target). The point oracle places a point at the voxel deepest inside that region, the scribble oracle a line along the middle of the region (the longest path through its skeleton) in the slice where the region is largest. Both use the same tools as the agents. We record the Dice after every prompt. As the agents prompt almost only with scribbles (Table 2), we compare each agent with the scribble oracle at the number of prompts the agent sent on each target (after five prompts for the 4–11% of targets where an agent sent more).

• VoxTell: no agent; VoxTell v1.1 [8] with exactly the target texts the agents get, all targets of an image in one call (for the 17 foot bones three per call, since VoxTell’s outputs for all 17 do not fit into the memory of our GPU; on another image, splitting the prompts changed its masks by less than 0.002 Dice).

	Prompts per target	Scribbles	Boxes	Points	From code	Views per target
Claude Code, guided	2.2	94%	4%	2%	14%	9.1
Claude Code, unguided	2.3	94%	3%	2%	17%	10.4
Codex, guided	2.9	100%	0%	0%	0%	13.9
Codex, unguided	2.8	99%	0%	1%	0%	14.3

Scoring. Each target is scored by Dice, using its object as it was when the agent last committed it; a later object that takes voxels from it (a tumour inside the liver) does not count against it. A missing target scores 0. Means are taken per image first: the targets of an image are averaged, and a track’s mean is the mean over the images (all 14, or those of one group), so that the image with 17 targets counts as much as an image with one.

Setup. nnInteractive ran locally with its public v1.0 weights on one NVIDIA RTX 4090. The agents were Claude Code 2.1 with Claude Opus 5.5 and Codex 0.162 with gpt-6.1-sol, both at medium reasoning effort. They worked through the nnInteractive tools, including a Python tool for their own code. Each agent track and the one-point track were run three times; the oracles and VoxTell once.

Results

The agents’ prompts are far better than a single click

With the request alone, the agents reach a mean Dice of 0.81 (Claude Code) and 0.84 (Codex); with one voxel per target as a hint, both reach 0.85 (Table 1). nnInteractive with one click per target at that same voxel reaches 0.71. The gain is largest where a single point cannot describe the target: on the liver of liver_vessels_ct, one point gives Dice 0.00 and the agents 0.95 to 0.96; on the liver tumour of hcc_ct, 0.13 against 0.96. Locating the targets costs the agents little: the unguided tracks are only 0.01 (Codex) and 0.03 (Claude Code) below the guided ones. No agent run left a target without an object.

Track	Dice				Prompts	Scribble oracle,	Minutes
	all (14)	radiology (9)	rad. without adrenal_ct (8)	legs, mouse, ant (5)	per target	same prompts	per image
Codex, guided	0.855 ± 0.002	0.823	0.812	0.912	2.9	0.869	3.8
Claude Code, guided	0.847 ± 0.006	0.828	0.818	0.880	2.2	0.844	3.6
Codex, unguided	0.842 ± 0.004	0.815	0.804	0.892	2.8	0.886	4.2
Claude Code, unguided	0.814 ± 0.006	0.814	0.803	0.813	2.3	0.854	4.3
One point, no agent	0.709	0.669	0.651	0.779	1	0.769	0.2
VoxTell, text only	0.539	0.777	0.875	0.110	–		1.5

With 14 images, these means carry considerable uncertainty: 95% intervals from a bootstrap over images are 0.11 to 0.14 wide for the agents, for example [0.79, 0.91] for Codex, guided. Comparisons on the same images are more precise. Every agent track is ahead of one click, by 0.11 [0.01, 0.22] (Claude Code, unguided) to 0.15 [0.05, 0.26] (Codex, guided). The location hint helps Claude Code (+0.03 [0.01, 0.07]) but not Codex significantly (+0.01 [−0.001, 0.027]). The two agents do not differ clearly: without the hint, Codex is ahead by 0.03, but the interval includes zero ([−0.003, 0.070]); with the hint, the difference is 0.01 [−0.01, 0.03]. The Dice of every target in every track is listed in Table S2 of the supplement.

Image	Target	Codex, guided	Claude Code, guided	Codex, unguided	Claude Code, unguided	One point	VoxTell	Scribble oracle, 5 prompts	Point oracle, 5 prompts
adrenal_ct	adrenal tumour	0.91	0.91	0.90	0.90	0.82	0.00*	0.93	0.91
ant_microct	brain	0.93	0.79	0.93	0.79	0.44	0.33	0.94	0.74
bladder_mr	bladder tumour	0.76	0.77	0.71	0.76	0.81	0.86	0.86	0.85
calf_mr	soleus	0.75	0.79	0.74	0.58	0.57	0.00	0.88	0.84
calf_mr	tibialis anterior	0.83	0.81	0.82	0.63	0.72	0.06	0.91	0.86
calf_mr	medial gastrocnemius	0.86	0.87	0.80	0.46	0.90	0.00	0.95	0.94
foot_ct	left talus	0.99	1.00	0.99	0.99	1.00	0.38	0.99	0.99
foot_ct	left calcaneus	0.99	0.99	0.99	0.99	0.99	0.06	0.99	0.99
foot_ct	left navicular	0.99	0.99	0.99	0.99	1.00	0.00	0.99	0.99
foot_ct	left cuboid	0.99	0.99	0.99	0.99	0.99	0.00	0.99	0.99
foot_ct	left medial cuneiform	0.99	0.99	0.99	0.99	0.99	0.00	0.99	0.99
foot_ct	left intermediate cuneiform	0.98	0.98	0.98	0.98	0.99	0.00	0.98	0.98
foot_ct	left lateral cuneiform	0.99	0.99	0.99	0.99	0.99	0.00	0.98	0.99
foot_ct	left metatarsal 1	0.99	0.99	0.99	0.99	0.92	0.00	0.99	0.99
foot_ct	left metatarsal 2	0.97	0.98	0.98	0.98	0.99	0.00	0.98	0.99
foot_ct	left metatarsal 3	0.99	0.98	0.99	0.98	0.99	0.00	0.98	0.98
foot_ct	left metatarsal 4	0.98	0.98	0.99	0.98	0.99	0.00	0.97	0.98
foot_ct	left metatarsal 5	0.99	0.98	0.98	0.98	0.91	0.00	0.99	0.99
foot_ct	left big toe phalanges	0.97	0.97	0.97	0.98	0.78	0.00	0.98	0.99
foot_ct	left toe 2 phalanges	0.96	0.97	0.97	0.96	0.88	0.03	0.98	0.98
foot_ct	left toe 3 phalanges	0.92	0.92	0.98	0.94	0.87	0.00	0.96	0.98
foot_ct	left toe 4 phalanges	0.97	0.96	0.97	0.96	0.86	0.00	0.98	0.97
foot_ct	left toe 5 phalanges	0.97	0.95	0.97	0.96	0.94	0.00	0.98	0.98
hcc_ct	liver	0.97	0.96	0.97	0.97	0.61	0.97	0.88	0.97
hcc_ct	liver tumour	0.96	0.96	0.96	0.96	0.13	0.94	0.96	0.97
headneck_mr	mandible	0.74	0.70	0.77	0.73	0.20	0.81	0.80	0.82
headneck_mr	brainstem	0.77	0.81	0.80	0.75	0.27	0.86	0.82	0.81
headneck_mr	left parotid gland	0.82	0.83	0.81	0.83	0.83	0.87	0.90	0.89
headneck_mr	right submandibular gland	0.80	0.79	0.80	0.80	0.80	0.85	0.89	0.85
headneck_mr	thyroid gland	0.68	0.68	0.66	0.66	0.48	0.76	0.77	0.72
headneck_mr	spinal cord	0.49	0.46	0.52	0.55	0.41	0.61	0.79	0.57
headneck_mr	left optic nerve	0.27	0.22	0.34	0.02	0.02	0.59	0.64	0.46
headneck_tumor_mr	primary tumour	0.79	0.79	0.80	0.78	0.74	0.88	0.88	0.88
liver_vessels_ct	liver	0.95	0.96	0.95	0.96	0.00	0.96	0.96	0.95
liver_vessels_ct	portal vein	0.41	0.47	0.42	0.50	0.35	0.53	0.49	0.47
liver_vessels_ct	hepatic veins	0.42	0.53	0.27	0.30	0.20	0.64	0.44	0.53
lung_ct	lung tumour	0.91	0.91	0.92	0.90	0.92	0.95	0.93	0.94
lung_pet	tumour	0.93	0.94	0.93	0.94	0.92	0.96	0.94	0.94
mouse_microct	tumour	0.91	0.89	0.83	0.82	0.91	0.00	0.96	0.96
pediatric_ct	liver	0.98	0.98	0.98	0.98	0.98	0.98	0.98	0.98
pediatric_ct	spleen	0.98	0.98	0.98	0.98	0.98	0.98	0.98	0.98
pediatric_ct	right kidney	0.98	0.97	0.98	0.97	0.97	0.98	0.98	0.98
pediatric_ct	urinary bladder	0.96	0.96	0.96	0.96	0.97	0.96	0.98	0.98
pediatric_ct	heart	0.93	0.92	0.91	0.90	0.42	0.96	0.93	0.93
pediatric_ct	pancreas	0.81	0.76	0.73	0.68	0.89	0.91	0.92	0.91
pediatric_ct	gallbladder	0.96	0.96	0.96	0.96	0.95	0.96	0.96	0.96
pediatric_ct	left adrenal gland	0.58	0.53	0.65	0.54	0.55	0.66	0.75	0.68
thigh_mr	rectus femoris	0.95	0.95	0.94	0.94	0.95	0.00	0.96	0.96
thigh_mr	sartorius	0.93	0.90	0.92	0.92	0.91	0.27	0.94	0.94
thigh_mr	gracilis	0.91	0.93	0.92	0.93	0.92	0.41	0.94	0.93
thigh_mr	vastus lateralis	0.90	0.87	0.92	0.91	0.70	0.00	0.95	0.94

The agents’ prompts are close to the oracles’; they stop earlier

To judge the quality of the agents’ prompts, we compare them with the scribble oracle at the same number of prompts: for every target, its Dice after as many prompts as the agent sent on it (Table 1, Figure 7b). With the location given, the scribble oracle is level with Claude Code (0.844 against 0.847; 95% interval of the difference [−0.03, 0.02]) and 0.015 ahead of Codex ([−0.01, 0.04]). Without the location, the scribble oracle is further ahead (0.04 for both agents): it knows where each target lies; the agents have to find it. After their first prompt, the guided agents reach 0.79, about as much as the scribble oracle after one prompt (0.77; the agents see their first scribble or box in a preview before sending it, whereas the oracle’s first scribble follows a fixed rule), and without the hint 0.74 (Codex) and 0.76 (Claude Code). The agents send 2.2 to 2.9 prompts per target, while the oracles keep improving with every further prompt (Figure 7a): the agents judge their results from rendered views and stop when a result looks right, whereas the oracles compare each result with the reference, whose exact extent a request in plain language cannot fully convey.

A text-prompted model is more accurate within its domain, but brittle

VoxTell segments all targets of an image in about 20 seconds from the target texts alone (17 minutes for the 17 foot bones in the whole-body CT) and reaches a mean Dice of 0.54. This average conceals two distinct regimes. On seven of the nine radiology images (and on 19 of their 25 targets) it is more accurate than every agent, by 0.02 to 0.09 per image, and it does well on thin structures that the agents find hard (left optic nerve 0.59 against 0.02 to 0.34 for the agents, spinal cord 0.61 against 0.46 to 0.55, hepatic veins 0.64 against 0.27 to 0.53); on hcc_ct it is level with them. On the adrenal tumour, however, it returns an empty mask, so its mean over the nine images (0.78) is below the agents’ (0.81 to 0.83; Table 1); without this image, it is 0.88 against 0.80 to 0.82. The empty mask is a matter of wording: “the tumor of the right adrenal gland”, the benchmark’s request, gives Dice 0, while “adrenal tumor” gives 0.79 (a check outside the benchmark). Several of the radiology source datasets are part of VoxTell’s training data (HaN-Seg, HNTS-MRG, RIDER Lung CT, HCC-TACE-Seg), so these numbers may be optimistic. On the second group VoxTell fails: it reaches 0.10 on the leg muscles (agents 0.74 to 0.87), 0.03 on the foot bones (agents 0.98), where five of its 17 masks are empty and most of the others mainly cover a different bone, and 0.17 on the mouse and the ant (agents 0.80 to 0.92). Its vocabulary of precomputed prompts names all of these muscles and each tarsal bone (the metatarsals and toe bones only as groups); the images, however, are far from standard radiology: fat-weighted research MRI, a whole-body CT of a cadaver and animals. This is the expected behaviour of a model that maps text to masks directly. The agent, in contrast, does not map text to masks: it recognises the target in the image and prompts nnInteractive.

Where the agents struggle

Thin and tubular structures are the hardest targets for the agents: the portal and hepatic veins (0.27 to 0.53), the left optic nerve (0.02 to 0.34) and the spinal cord (0.46 to 0.55). They are hard to prompt, since a scribble must follow a narrow structure across slices, and hard to judge from renderings. Per-image results for all tracks are in Figure S2 and Figure S3–Figure S4 of the supplement, per-target results in Table S2. Without the location hint, Codex is ahead of Claude Code (0.84 against 0.81), mostly on the calf muscles and the ant brain; on this small benchmark, the difference is not clearly significant (§ 6.1).

How the agents work

Both agents follow the skill: almost all prompts are scribbles, boxes and points are rare (Table 2). Claude Code sends about a sixth of its prompts from its own Python code; Codex prompts only through the tools but looks at more views, including four to six times as many 3D renderings. An image takes about four minutes (3.6 targets on average). At API prices, Claude Code’s runs cost $1.29 (guided) and $1.46 (unguided) per image ($115 for both tracks together), mostly for 2.8 to 3.1 million cached input tokens; Codex used about 1.7 to 1.9 million input tokens per image, 96% of them cached. The agents’ notes are informative: objects whose commit note expresses uncertainty (“approximate”, “uncertain”, . . . ) reach a mean Dice of 0.72 (Claude Code) and 0.68 (Codex), against 0.92 and 0.93 for the others.

Discussion

An extension, not a new agent. nninteractive-agent adds a capability to existing agents rather than building a new one. This has three consequences. First, it is not tied to one model: the same tools already serve two commercial agents and an open model, and we expect a better model to become a better annotator without any change on our side. Second, the agent keeps its general abilities (anatomical knowledge, coding, web search, conversation), on which the examples in § 4 rely and which a specialized model fine-tuned for promptable segmentation tasks does not provide to the same extent. Third, segmentation becomes one tool among others in the agent’s usual environment: it needs no dedicated viewer, its results can feed into whatever else the agent does with the data, and further tools, such as a text-prompted model, a detector or a registration tool, could be added in the same way. The other side of this is a dependence on the agents’ providers, both for the agents’ quality and for which data may be sent to them (see Limitations).

What nninteractive-agent enables. With nninteractive-agent, an existing agent takes over the part of interactive segmentation that a promptable model leaves to its user. Told where a target is, it prompts nnInteractive almost as effectively as a simulated user who knows the answer, at the same number of prompts, and without that hint it loses little. More importantly, it handles requests that no fixed interface anticipates: counting lesions it first has to find, muscles described by their function, two dozen ribs that have to be numbered. It brings anatomical knowledge, plans the work, writes code where code is the better tool, checks its results, and says what it is unsure about. These abilities are the agent’s own; nninteractive-agent, a set of tools and a skill, connects them to nnInteractive without training a model.

Agentic versus text-prompted segmentation. The comparison with VoxTell shows the trade-off. A text-prompted model is fast and, on most standard radiology targets, more accurate than the agents. It is also brittle: a request worded differently from what it expects can return an empty mask, and on images far from standard radiology (single muscles in fat-weighted MRI, the bones of the foot, animals) it fails. The agent needs to recognise the target in the image and to prompt nnInteractive, which is not tied to a fixed set of structures; it therefore keeps working where the text-prompted model stops, at the price of minutes per image. The two are complementary. A text-prompted model could be one more tool for the agent, giving a first draft for standard targets that the agent checks and corrects with nnInteractive; we leave this for future work.

Limitations. The agents stop earlier than an oracle would; better self-assessment is the most direct way towards oracle performance. Thin and tubular structures remain difficult to prompt and to judge from renderings. A run takes minutes and, at API prices, costs $1.3 to $1.5 per image, which suits annotation and research use better than the processing of large cohorts. We did not compare the agents with human annotators. An experienced nnInteractive user would likely be more accurate, particularly on thin structures, and faster per target; the agent’s advantage is that nobody needs to sit at the screen. With a commercial agent, rendered views of the image are sent to the LLM provider, which has to be compatible with the terms of use of the data; open models served in-house avoid this; we tested one only as a proof of concept (§ 3.4). The benchmark is small (14 images, 51 targets), the oracles follow simple rules and show what well-placed prompts achieve rather than an upper bound. The nnInteractive and VoxTell weights are licensed for non-commercial use only.

Conclusion

General-purpose AI agents can act as 3D image annotators: with a set of tools and a skill, and without additional training, they operate nnInteractive much as human annotators do. They place prompts in slices, check the 3D result and correct it. From plain-language requests they reach a mean Dice of 0.81 to 0.84 on a diverse benchmark, and 0.85 when told where each target is, prompting nearly as effectively as an oracle given the same number of prompts; keep working on data where a text-prompted model fails. Beyond prompting, they plan, write their own code and take corrections in plain language. As agents improve, we expect this way of annotating images to improve with them.

Code availability. nninteractive-agent, comprising the MCP server, the skill, the web demo, the benchmark scripts and the aggregated benchmark results (per track and per image): https://github.com/MIC-DKFZ/nninteractive-agent (Apache-2.0). nnInteractive: https://github.com/MIC-DKFZ/nnInteractive.

Data availability. All images used in this work, except the suitcase CT scan in Figure 1 (provided by Smiths Detection), are available from their original sources, the thigh and calf MRI on request; we do not redistribute any image or reference mask. The benchmark sources, their licences and references are listed in Table S1. The build script (benchmark/prepare.py) downloads and converts them, derives the reference masks as defined in benchmark/cases.json and verifies the result against published checksums. Some sources require a free registration (VEELA) or a manual download in a web browser (ant brains). The thigh and calf MRI is no longer in its figshare record and is available from its authors on request.² The Visible Human images are courtesy of the U.S. National Library of Medicine. The examples in § 4 use MSD Liver (cases 51 and 155; CC BY-SA 4.0) [30, 31], TotalSegmentator MRI v2.0.0 (case s0506; CC BY-NC-SA 2.0) [1] and RibFrac (case 100; CC BY-NC 4.0) [2, 3]; the open-model test in § 3.4 uses MSD Spleen (case 10; CC BY-SA 4.0).

Acknowledgements. This work was supported by the NCT Heidelberg Image Data Analysis Infrastructure (IDAI) and by the Helmholtz Association under the Helmholtz Foundation Model Initiative (HFMI), project “The Human Radiome Project” (THRP). MR was supported by the Google PhD Fellowship Program. BH was funded by “NUM 3.0” (FKZ: 01KX2524) and by the Vector Stiftung. PV is partially funded by an Else Kröner Clinician Scientist Endowed Professorship from the Else Kröner Fresenius Foundation (reference number: 2022_EKCS.17).

Supplementary material

Table S1: Benchmark sources. The build script downloads these sources and derives the reference masks as defined in benchmark/cases.json.

(a) Instructions

You segment images with the nninteractive tools. Before the first nninteractive tool call, read the nninteractive-segmentation skill and follow it. Work autonomously: there is no human to answer questions during the run.

(b) Guided request (adrenal_ct)

This is an abdominal CT scan: <path>/adrenal_ct/image.nii.gz.
Segment each of the following structures as its own object, named exactly by its id. For each, a voxel inside it is given ([axis0, axis1, axis2], the coordinates your tools use).
Work through them in this order and commit each object before you start the next one:
- t01_adrenal_tumor: the tumor of the right adrenal gland, contains voxel [74, 295, 165]
Save the result when you are done.

(c) Unguided request (headneck_mr)

This is a T1-weighted MRI of the head and neck: <path>/headneck_mr/image.nii.gz.
Segment each of the following structures as its own object, named exactly by its id.
Work through them in this order and commit each object before you start the next one:
- t01_mandible: the mandible
- t02_brainstem: the brainstem
- t03_left_parotid_gland: the left parotid gland
- t04_right_submandibular_gland: the right submandibular gland
- t05_thyroid_gland: the thyroid gland
- t06_spinal_cord: the spinal cord
- t07_left_optic_nerve: the left optic nerve
Save the result when you are done.

(d) VoxTell’s prompts (headneck_mr, one call)

"the mandible" "the brainstem" "the left parotid gland" "the right submandibular gland"
"the thyroid gland" "the spinal cord" "the left optic nerve"