WorksComputer Science
AffordJEV: Affordance Grounding by Confirming Candidate Regions
Abstract
Affordance grounding asks which visible part of an object performs a requested operation, a basic capability for embodied manipulation and human–robot interaction. Most existing methods follow a System Two paradigm: a multimodal language model first writes a chain of thought and then turns its conclusion into a box or points for a segmenter. This incurs considerable inference cost, gives the semantic decision little direct control over the extent of the final region, and makes mistakes hard to correct. Yet locating an affordance is essentially a perceptual judgment that need not be argued out in words. We therefore propose, in which frozen segmenters propose candidate regions and a System One model decides directly which of them perform the operation, generating no text and reading each decision from a single forward pass. Our contributions are twofold. (1) An inference framework that confirms candidate regions level by level, from object to part to sub-part, with evidence passed between the levels in both directions so that the levels correct one another. (2) A System One model trained for this framework: existing System One models are built for general-purpose decisions, judge one candidate per request and are ill-suited to the framework's region questions, so ours learns directly from the requests the framework itself produces and judges all candidates of a level at once. Trained only on InstructPart and evaluated with definition-aligned operation words, outperforms chain-of-thought methods in domain and in zero-shot transfer to UMD and IIT-AFF, is better overall than existing System One models run in the same framework, and needs fewer judge requests. To our knowledge, is the first approach to bring System One models to affordance grounding.
Introduction
A robot asked to cut with a knife needs the blade, not merely a correctly recognized knife. Affordance grounding connects an object and an intended operation to the visible surface that performs it. The output must be spatially precise, but the
difficulty is mostly semantic: a handle, a working surface, and the support under an object are all clean segments, and only one of them serves the requested operation.
A popular recipe is to let a multimodal language model reason about the request before answering. Affordance-R1 [42] and VisionReasoner [26] write a reasoning trace, decode a box or points, and hand them to a segmenter. This is a System Two procedure in the sense of Kahneman [17]: slow, sequential, and general. Yet grounding an affordance rarely calls for deliberation. People do not need to think in words to find the handle of a mug; they look at the mug and then at its parts. This suggests a System One alternative in which segmenters propose regions and a judge scores them directly, reading its decision from a single forward pass instead of decoding text. The alternative is already competitive: placed in our region pipeline without any fine-tuning, the released Valen-4B judge [40] reaches 75.0% gIoU on UMD [29], against 63.9% for Affordance-R1 under aligned operation words (Tab. 2 and Figure 1).
Figure 1. Results on InstructPart, UMD and IIT-AFF (%). Each axis is one metric, scaled to the range of the four methods; the numbers are our scores (Tabs. 1 to 3).
Such a pipeline must still decide what to ask the judge. One question over every region of an image is fragile: the queried part is small, regions of other objects compete with it, and a wrong object is never corrected. People narrow down instead, and they revise: if no part of the chosen object fits the operation, the object was probably wrong. Our first contribution turns this into an inference framework. The judge confirms the object, selects its parts and refines them with sub-parts, and its verdicts pass between the levels: weak part evidence under an unsure object reopens the object decision, and sub-part verdicts trim an accepted part or rescue a rejected one. Every step asks a question of the same form — which of these regions answer the query — so one judge serves all levels.
Within this framework, what limits accuracy is not the candidates but the interface through which they are judged. On the regions our frozen segmenters propose, a judge that accepted exactly the candidates lying mostly inside the ground truth would reach 77.6 gIoU on InstructPart; the released Valen-4B extracts 51.5 from the same regions, and the strongest System Two model reaches 59.5. The judges used for this are general-purpose decision models built for arbitrary multiple-choice questions: they see one candidate per request, learn its identity from a marker painted into the image, pool the whole context into a single vector in which a small part is diluted, and score through a head trained from scratch on millions of unrelated decisions. None of these choices follows from the task.
Our second contribution is a judge built for the framework’s questions. Each candidate becomes one token, pooled from the vision encoder’s own output over its mask and given its geometry, so the judge reads the pixels a candidate covers instead of a drawn marker. All candidates of a level share one sequence and one forward pass, and a small set block lets them see one another, because the framework outputs the union of what it accepts. The score is the projection of the readout state on the model’s own Yes−No embeddings, which adds no parameters and lets the judge be trained from the raw Qwen3.5-4B weights [33]: before any training it already separates parts at AUROC 0.618. We train it on the candidates the frozen segmenters produce on the training photos, at every level the framework asks about, with each candidate’s purity as the target that acceptance at 0.5 needs.
Comparing judges fairly required care. We retrained Valen and JevAny with their own native heads and losses on our object and part groups, from the same raw weights and with matched optimisation settings. We also found that the same operation word names different regions in different datasets — InstructPart marks a vessel’s body for grasp, UMD and IIT-AFF a tool’s handle — so a judge that fits the training convention better is penalised for it on transfer. We therefore report a frozen definition-aligned protocol alongside the original words.
Our contributions are:
An inference framework that grounds affordances by confirming candidate regions level by level — object, part, sub-part — with evidence passed between the levels in both directions, so that one judge answering one form of question serves the whole hierarchy. Removing evidence passing lowers in-domain gIoU from 70.4 to 65.7.
A System One model for this framework, the region-set judge, trained from raw weights on about thirty thousand framework requests: candidates as pooled region tokens, a level judged in one forward pass with the candidates visible to each other, and decisions read through the model’s own answer embeddings. It leads the retrained native-head baselines on eleven of twelve aligned columns with four times fewer requests, and each ablated component costs at least one point of in-domain gIoU.
Related Work
Affordance grounding. Affordances describe what an object offers to an agent [11]; early supervised models segment them per pixel [10]. UMD pairs tool parts with functional labels [29], and IIT-AFF annotates affordances on objects in cluttered scenes [31]. HO-3D-AFF and CCM-AFF move evaluation to hand-held containers under occlusion [1, 2, 3], InstructPart provides part-level masks for task instructions in everyday photos [41], and AGD20K annotates interaction heatmaps [27]. Weaker supervision comes from human interaction: hotspots learned from video [30], part transfer from exocentric to egocentric views [20], and interaction itself for articulated objects [28]. OOAL adapts foundation features from sparse labels [21], and recent work probes the geometric and interaction cues that such features carry [47]. AffordanceLLM brings vision–language knowledge to the problem [32]; Affordance-R1 trains reasoning-based grounding with reinforcement learning on data derived from InstructPart [42]; A4-Agent and AFUN pursue agentic and foundation-model formulations [43, 49], and ThinkAfford reasons about affordances in cluttered 3D scenes [24]. We take the opposite direction to reasoning: the model makes direct decisions on proposed regions, and the decisions are organized across levels.
Segment, then decide. SAM and PartGLEE produce transferable object and part masks [18, 22, 35], Grounding DINO detects objects named in text [25], and SAM 3 segments all instances of a named concept [8]. Part datasets and segmenters [13, 34, 38] define parts by name, not by the operation they serve. Set-of-Mark prompting lets a multimodal model refer to marked regions [45], LISA couples a language model with a mask decoder [19], and Affordance-SAM connects promptable segmentation to affordances [16]. VisionReasoner unifies several perception tasks behind a reasoning stage [26]. These systems usually make one decision over one set of regions. AffordJEV instead makes a sequence of decisions over a hierarchy of regions and lets later decisions revise earlier ones. Coarse-to-fine cascades in detection refine a hypothesis stage by stage [6]; in our hierarchy, evidence also flows back up and can replace the hypothesis.
Reasoning and direct decisions. Chain-of-thought prompting [44] and reinforcement learning on verifiable rewards [9] let language models reason before answering, and Affordance-R1 and VisionReasoner carry this recipe to grounding. A second line uses models as judges of candidate answers rather than as generators [23, 50]. Valen exposes choice and yes/no decisions of a multimodal backbone through a role-based scoring head [40], and several 4B-scale models offer similar direct interfaces [4, 12, 37, 39]. These are general-purpose decision models: one candidate per request, candidate identity supplied by a drawn marker, and a head trained on large collections of unrelated decisions. We refer to such models, which answer in one forward pass without generating text, as System One models. We keep the idea of a direct judge and replace the interface with one specific to region confirmation.
How a decision is read out. Reading a classification from the token an autoregressive model would have produced — a verbalizer — is standard in prompt-based classification [5, 36] and is why few-shot prompting works at all without a new head. We use it as the judge’s scoring function rather than as an inference trick: the score is a projection of the readout state on the model’s own answer embeddings, which keeps the pretrained notion of yes available and is what permits training on about thirty thousand requests instead of millions. Passing a region to a language model by pooling encoder features over its mask, instead of cropping or painting it, follows region-feature practice in detection and segmentation [14, 18] and in region-level multimodal models [46, 48]; we additionally let the regions of one level attend to one another, which is natural here because the framework outputs their union, so a candidate’s correct label depends on which others are accepted. Set-level prediction of this kind is familiar from detection with set losses [7], though there it removes duplicates rather than composing an answer.
Method
Given an RGB image , an object name and an operation , AffordJEV predicts a binary mask on the image grid; no box, point or example is given at test time. The method has two parts designed together. The framework (§ 3.1) turns a query into a short sequence of questions, each asking which regions of a candidate set answer it. The judge (§ 3.2) is built for exactly these questions and trained on the requests the framework itself produces (§ 3.3). Figure 2 shows both on one real query.

Figure 2. AffordJEV. Top: one test query through the three levels. At every level the same judge scores all candidates at once (bars, threshold 0.5); a weak part verdict can reopen the object level (blue) and a part is split into sub-parts (orange). Bottom: the judge on one request (§ 3.2); colors follow each candidate from its mask to its score.
The framework: a hierarchy of region questions
Questions and decisions. Every request has the same form: a level , an image or crop, candidate masks and the query , for which the judge returns one logit per candidate. At the selection levels, part and sub-part, a candidate is accepted when , and a foreground yields the union of what is accepted together with the strongest verdict, which measures how well is supported:
At the choice level, the object extent, exactly one candidate is right, and the probabilities are a softmax over the logits of the group. Every judge we compare against is used through this interface and these rules.
Object level. The query names the object, so object candidates come from SAM 3 [8] prompted with . When instances reach the detector’s own threshold of 0.5, their union, with holes filled so that an object also covers the region it encloses, such as a bowl’s interior, is the object, and the object decision counts as settled. Otherwise the three highest-scoring distinct masks (pairwise IoU at most 0.9) form a choice question — which mask covers the complete object? — and the largest probability states how sure the object decision is. Behind these hypotheses the ranking continues with class-agnostic regions from PartGLEE [22], resolved into mutually exclusive pieces and visited only by backtracking.
Part and sub-part levels. For a foreground we crop around with 10% padding and run PartGLEE with the generic text part on the crop with its exterior grayed out; the judge later sees the crop without the gray. Proposals are clipped to , given disjoint pixels and capped at six; discarded small proposals are merged into connected clusters, and pieces of that no candidate covers become candidates of their own. SAM 3 additionally receives the part names that the training data lists for , such as cup handle, and a named part lying mostly inside takes precedence over the generic parts it overlaps. The same call accepts any mask as foreground, so a part can be split once more into sub-parts clipped to ; since the crop shrinks with the region, a sub-part is seen at a higher effective resolution than its parent.
Passing evidence across levels. The levels are not visited independently. Upward, a weak part verdict questions the object: if the object decision is unsure () and its foreground is not confidently supported (), with , the search visits the next object hypotheses in order, up to , and keeps . Downward, every part of is split and the sub-part verdicts refine the part decision: if no part is accepted although the object is settled, the confirmed sub-parts become the prediction; if a part is accepted, its rejected sub-parts are cut out of it, while a uniform verdict rejects it whole. The judge is therefore asked three kinds of question — choose the object extent, select parts, select the sub-parts of one part — and every answer is a subset of a candidate set.
The region-set judge
These questions have three properties that general-purpose judges [12, 37, 40] do not use. The answer depends on the pixels a candidate covers, often a small part of the image. It is a property of the set, because the framework outputs the union of what it accepts. And the framework can supply about thirty thousand training requests, too few to train a new scoring head from scratch. We build one judge around these properties, starting from the raw Qwen3.5-4B weights [33].
Request layout. The object name and operation come first, followed by a fixed sentence that names the level, such as The image is a crop around the ; each candidate region is one part of the ; only and are filled in. The image follows, so that its visual tokens are encoded conditioned on the query, then a question asking Yes or No for each region, and the candidates as rows Region k: Rk Answer:. A sub-part request also carries the token of the parent part. Candidate is read at the final colon of its row.
Regions as tokens. Candidate enters the sequence as one token whose embedding is the coverage-weighted mean of the vision encoder’s own output over the token cells its mask touches, which already lives in the language model’s input space, plus a zero-initialised embedding of its box, area and centroid:
The judge therefore reads the pixels a candidate covers rather than inferring them from a drawn marker. Region tokens sit at text positions after the image, so where a candidate lies and how large it is enter only through .
A level judged as a set. All candidates of a level share one sequence and one forward pass. This replaces one request with three images per candidate by one request with one image per level: over the test queries the judge answers 8.1 requests and encodes 8.1 images per query instead of 33.0 and 99.1. Inside the causal language model a candidate sees only those listed before it, and a small set block makes the comparison symmetric. For each candidate it takes the readout state together with , the last-layer image features pooled with the same weights , adds one token for the final position of the sequence, and passes them through two bidirectional Transformer layers without positional encoding. Its zero-initialised output is a residual logit ; mixing is across the candidates of one request, never across levels.
Reading the decision from the answer embeddings. The score of candidate is
where and are the model’s own tied output embeddings for the two answer tokens. The first term is a scaled log-odds of answering Yes rather than No at the readout position. It adds no parameters and is meaningful before any training: the untrained model separates part candidates at AUROC 0.618 and picks the object extent in 61.0% of groups, against 33.3% chance. This is what makes training from raw weights on so few requests possible. Selection levels use and the extent choice a softmax, as in Eq. (1).
Training on the framework’s own requests
The judge must decide on the candidates the framework actually produces — fragments, mixed regions and wrong foregrounds — and its probabilities must mean what the decision rule assumes. Training follows from these two requirements.
Requests from the framework. We run the frozen segmenters on the InstructPart [41] training photos and build requests at every level the framework asks about: 1,132 object-extent choices from SAM 3 prompted with the object name, labelled by the mask that best matches the reviewed object and kept when that match reaches IoU 0.5; 5,584 part groups; and 19,081 sub-part groups, resampled to 8,000 per epoch. Two auxiliary tasks are never asked at test time and only shape the shared representation: 2,592 object-ownership choices among class-agnostic regions, and 2,344 part-name groups that ask which candidates form a named part, such as the handle of a fork. In total the judge is trained on 30,733 candidate groups.
Targets that match the decision rule. A candidate is accepted when , so should estimate how much of the candidate belongs to the answer. We therefore supervise it with the candidate’s purity, the fraction of its reliably labelled pixels inside the queried part, through binary cross-entropy, together with a second binary cross-entropy on its membership in the IoU-optimal union over all subsets of the group. Candidates with less than half of their pixels reliably labelled are left out of the loss, and choice requests use cross-entropy on the softmax over the group. Optimising the expected IoU of the union directly, which seems to match the framework’s output more closely, is worse (§ 4.3).
Optimisation and calibration. Starting from the raw weights, we train LoRA adapters [15] in the language backbone together with the visual merger, the geometry embedding and the set block for two epochs, clipping gradients per parameter group because the merger and geometry gradients are two orders of magnitude larger than the adapter ones. 217 photos that share no photo group with the training photos are held out, and on them we fit one temperature for the selection levels and one for the choice level. The InstructPart evaluation queries are drawn from these photos. A temperature leaves acceptance at 0.5 and the extent argmax unchanged but enters the backtracking gate. No evaluation image is used for training or model selection.
Experiments
Setup
Benchmarks. All test sets are stratified by operation, with up to ten queries per operation drawn with a fixed seed before any prediction was seen. InstructPart [41] contributes 186 queries over 38 operations from 126 held-out photos that share no photo group, file or decoded image with the training photos; they are among the held-out photos of § 3.3, so this is an in-domain validation set, not the official test split. UMD novel/test [29] contributes 70 queries on 70 images over seven operations, with void pixels ignored, and the official IIT-AFF test split [31] 90 queries on 88 images over nine operations, restricted to queries whose target pixels belong to a single object category. UMD and IIT-AFF are zero-shot for every method. gIoU averages per-query IoU, cIoU divides the summed intersection by the summed union, is the fraction of queries above IoU 0.5, and averages that fraction over thresholds 0.5 to 0.95.
Definition-aligned evaluation. The same operation word does not name the same region in every dataset. InstructPart marks a vessel’s body for grasp, while UMD and IIT-AFF mark its handle; support marks the load-bearing surface of furniture in one and the blade of a shovel in the other. A judge that learns the training convention well is therefore penalised on the transfer sets for doing so, which measures annotation style rather than grounding. We report both protocols. Under original words the query is passed verbatim. Under aligned words the operation text of the conflicting classes is replaced through a mapping fixed before evaluation and derived from the native annotations alone, with no model scores or metric search (Table 5): on UMD and IIT-AFF grasp→hold, wrap grasp→grasp, and contain→contain in the hollow interior or opening. For support, UMD shovels use dig and IIT-AFF spatulas use support food on the flat working blade. It touches 80 of 346 queries and no InstructPart query; images, ground truth, valid pixels, candidate pools and decision rules are untouched.
| Word | Test sets mark | InstructPart marks | Aligned to |
| grasp | tool handle | vessel body | hold grasp |
| wrap grasp | vessel body | — | grasp |
| contain | hollow interior or opening | body, interior or opening | contain in the hollow interior or opening |
| support | UMD: shovel blade | furniture seat or surface | dig |
| support | IIT-AFF: spatula blade | furniture seat or surface | support food on the flat working blade |
Table 5. Definition alignment on UMD and IIT-AFF.
Baselines. Affordance-R1 [42] and VisionReasoner-7B [26] run from their released weights through their official inference paths. The released System One judges Valen-4B [40], H2O-Lightning-4B [12] and JevAny-4B-LoRA [37] are placed in our pipeline with the same candidates and rules. Because those judges were pretrained on millions of unrelated decisions, we additionally retrain Valen and JevAny with their own native heads and losses on our object and part supervision, from the same raw Qwen3.5-4B weights, with matched LoRA rank, learning rates, optimiser, seed and epoch budget; these are the directly comparable baselines and we mark them retrained. Every method receives the same RGB, object name and operation.
Main results
Tables 1 to 3 report all three sets under the aligned operation words. The rows that matter most are the two retrained baselines: they share our object and part groups, base weights, optimiser settings, two-epoch budget, candidate pools and decision rules. Our full judge additionally uses sub-part, extent and part-name supervision and purity targets, so this compares complete training schemes. AffordJEV leads them on eleven of the twelve columns. The margins are largest where the candidates are hardest to read — +3.80 gIoU and +5.85 cIoU on UMD, +4.68 in domain — and the one exception is cIoU on IIT-AFF, where the retrained JevAny is ahead by 0.47. cIoU pools pixels over queries, so it is decided by a few large-area contain regions on which our judge accepts more of the vessel than the annotation marks; § 4.5 identifies them.
Across all eight methods, we lead ten of twelve columns. Both System Two methods trail on all twelve columns, Affordance-R1 by 11.0, 11.4 and 7.0 gIoU on InstructPart, UMD and IIT-AFF. Of its 124 queries below IoU 0.5, 38 already cover at least 70% of the target but predict at least twice its area: the decoded box or point lands in the right place, and the segmenter decides the extent. Released Valen reaches 74.97 gIoU on UMD, up from 62.35 under original words, and wins with 90.00 against our 85.71; in domain it trails by 18.90 gIoU. Released H2O, released JevAny and VisionReasoner reach 68.36, 59.09 and 54.79 gIoU on UMD, respectively, and trail our judge on every column of both transfer sets.
The alignment is what makes these rows comparable, and the size of its effect is worth stating. Our judge gains 14.22 gIoU on UMD when the conflicting words are aligned, while the retrained JevAny gains 6.39. The unequal gains show sensitivity to the operation words; they do not isolate an architectural cause. One query makes this concrete. On ladle_03/00000223/grasp our judge selects the bowl of the ladle under the original word and the handle under the aligned word; the aligned prediction is byte-identical to the retrained JevAny’s original-word prediction, and the selected object region never changes. Original-word numbers for every method are in the supplementary material.
Ablation
The framework. Both framework ablations keep the full judge and change only inference. Without evidence passing each level is decided once, top-down, with no backtracking and no sub-part refinement. Under the aligned protocol this lowers gIoU from 70.44 to 65.69 in domain and from 75.26 to 73.57 on UMD, and by 7.5 and 4.3 points; on IIT-AFF the framework is slightly better without it (68.72 against 67.92 gIoU), so the gain is not uniform. Without the hierarchy, PartGLEE proposes parts on the whole image and the judge answers a single part question: InstructPart gIoU falls to 40.93, and 65 of the 186 queries end with no accepted region.
| Method | gIoU↑ | cIoU↑ | ↑ | ↑ |
| Affordance-R1 [42] | 59.49 | 60.93 | 46.51 | 64.52 |
| VisionReasoner-7B [26] | 47.26 | 42.01 | 31.94 | 48.92 |
| Valen-4B, released [40] | 51.54 | 51.35 | 36.67 | 54.30 |
| H2O-Lightning-4B, released [12] | 46.96 | 48.10 | 30.91 | 49.46 |
| JevAny-4B-LoRA, released [37] | 35.10 | 33.91 | 17.74 | 32.80 |
| Valen, retrained | 65.93 | 70.61 | 53.66 | 72.04 |
| JevAny, retrained | 67.67 | 72.72 | 54.84 | 74.19 |
| AffordJEV | 70.44 | 72.84 | 59.52 | 78.49 |
caption: tbl:1. InstructPart (%), in domain. The alignment of § 4.1 touches no InstructPart query, so these are both protocols at once. Released are the published judges in our pipeline; retrained are their native heads trained on our supervision.
:::
| Method | gIoU↑ | cIoU↑ | ↑ | ↑ |
| Affordance-R1 [42] | 63.89 | 62.43 | 43.00 | 64.29 |
| VisionReasoner-7B [26] | 54.79 | 55.19 | 22.14 | 48.57 |
| Valen-4B, released [40] | 74.97 | 75.49 | 58.71 | 90.00 |
| H2O-Lightning-4B, released [12] | 68.36 | 71.87 | 49.14 | 84.29 |
| JevAny-4B-LoRA, released [37] | 59.09 | 46.03 | 35.86 | 62.86 |
| Valen, retrained | 71.46 | 72.88 | 52.86 | 81.43 |
| JevAny, retrained | 71.11 | 71.68 | 51.29 | 78.57 |
| AffordJEV | 75.26 | 78.73 | 59.86 | 85.71 |
caption: tbl:2. UMD novel/test, zero-shot (%), under the aligned operation words of § 4.1. The alignment is applied identically to every method.
:::
| Method | gIoU↑ | cIoU↑ | ↑ | ↑ |
| Affordance-R1 [42] | 60.94 | 68.55 | 42.78 | 63.33 |
| VisionReasoner-7B [26] | 53.27 | 65.81 | 32.78 | 50.00 |
| Valen-4B, released [40] | 57.16 | 66.78 | 38.00 | 57.78 |
| H2O-Lightning-4B, released [12] | 54.02 | 60.82 | 33.22 | 57.78 |
| JevAny-4B-LoRA, released [37] | 55.98 | 63.38 | 34.22 | 62.22 |
| Valen, retrained | 66.43 | 73.05 | 49.22 | 75.56 |
| JevAny, retrained | 67.33 | 73.20 | 50.11 | 74.44 |
| AffordJEV | 67.92 | 72.73 | 52.78 | 77.78 |
caption: tbl:3. IIT-AFF test, zero-shot (%), aligned operation words. cIoU pools pixels over queries and is the one column we do not win; § 4.5 identifies the queries responsible.
:::
The judge. Table 4 removes one element of the judge or of its training at a time. Every row is a retrained judge except one candidate per pass, which re-scores the full judge with one candidate per request. In the head, region tokens are the largest term (−2.86 gIoU); this row also disables the answer-embedding readout, which has nothing candidate-specific to read without them, and removing only their geometry embedding costs 2.55. Judging one candidate per request costs 1.29 on top of being 4.1× more expensive, removing the set block 1.06, and placing the query after the image 1.45. In training, the framework’s own candidates dominate: replacing them with clean ground-truth parts costs 12.28 gIoU and 20.60 P, partly because ground truth offers no sub-parts to train on. Extent requests (−2.81), sub-part requests (−1.96) and purity targets (−1.60) follow; part-name supervision costs 1.20 gIoU but gains 1.51 cIoU. Every row lies above the 0.02 variation of re-scoring one model and the 0.70 spread among configurations, though each is a single training run.
| Variant | gIoU↑ | cIoU↑ | P↑ | P↑ |
| AffordJEV | 70.44 | 72.84 | 59.52 | 78.49 |
| Framework | ||||
| A1 Flat judgement | 66.76 | 70.32 | 55.05 | 72.04 |
| A2 w/o evidence passing | 65.69 | 68.23 | 54.30 | 70.97 |
| Judge architecture | ||||
| B1 Drawn region markers | 65.57 | 73.03 | 52.85 | 70.97 |
| B2 Per-candidate judgement | 67.53 | 65.60 | 56.45 | 73.66 |
| B3 w/o Yes/No readout | 69.23 | 74.45 | 57.04 | 77.96 |
| Judge training | ||||
| C1 GT part candidates | 68.53 | 73.43 | 55.11 | 74.19 |
| C2 Part-level training only | 65.52 | 70.11 | 52.26 | 69.89 |
| C3(a) Membership target only | 68.84 | 70.99 | 56.99 | 75.27 |
| C3(b) Purity target only | 68.38 | 74.20 | 57.58 | 74.73 |
Table 4. Ablations on InstructPart (%), grouped by framework, judge architecture and training. All rows use and one training seed. A1–A2 change inference only; B1–C3(b) are retrained. Bold marks the best value in each column.

Figure 3. Qualitative comparison against the two retrained baselines, all under the aligned protocol. Prediction purple, ground-truth contour green; numbers are IoU. The six left columns are the largest margins in our favour on each set, one query per object and operation. The last column is the largest margin against us: a second suitcase whose telescopic handle our judge rejects entirely while both baselines accept it.
Objectives shaped like the pooled metric. Adding an expected-IoU objective, which maximises the expected IoU of the union over the subsets of a group and seems to match the framework’s output exactly, costs 1.36 gIoU, 2.05 P and 3.76 P while gaining 0.43 cIoU: it rewards accepting an uncertain, mostly correct candidate, trading per-query accuracy for pixel recall. Weighting the purity loss by candidate area behaves the same way, raising cIoU to 74.09 and losing the other eleven columns. On a configuration that still contained the expected-IoU term, six of the nine elements above fell within gIoU, so an ablation run on a model with a harmful component understates the rest.
Ablations and the transfer sets. The ablated judges were scored with the original operation words only, and there the operation-word conflict of § 4.1 dominates: several of them score higher on UMD than the full judge (71.4 without purity targets, 70.0 without part-name supervision, against 61.0), the same convention conflict listed in Table 5. We therefore draw no conclusion about the ablations from the transfer sets.
Implementation
The judge starts from the published Qwen3.5-4B weights, with LoRA rank 32 (, no dropout) on every linear layer of the language tower, a trainable visual merger, and the new geometry embedding and set block; 101M parameters are updated. Learning rates are for the adapters, for the merger and for the new modules, AdamW with weight decay 0.01, 100 warm-up steps and a cosine schedule floored at 2%. Each of two epochs sees all object, extent, part and part-name groups and a fresh sample of 8,000 sub-part groups; one group is one optimiser item, with gradients accumulated over four and clipped per parameter family. Images are capped between and pixels, candidate order is shuffled, and crops are flipped with probability one half. Training takes about 70 minutes on seven RTX 4090 GPUs. The retrained baselines use their own native heads, compilers and losses with the same seed, rank, learning rates, optimiser, clipping and epoch budget on the object and part groups, which is the supervision their interfaces accept. Valen trains its head alone for one epoch and jointly for one; JevAny and our judge update LoRA for both epochs. The full judge sees more training groups, so matching epochs does not match optimiser steps or compute. Released H2O and JevAny recompute the 80 changed queries; the other 266 masks are unchanged. VisionReasoner reuses its full-test mapped predictions after verifying the model, mapping and image/annotation identity. All three evaluations cover 346 queries without inference failures.
Analysis
Where the judge still loses. The candidates admit far more than any judge extracts: an oracle that accepts exactly the candidates lying mostly inside the ground truth reaches 83.22 cIoU in domain and 85.24 / 85.63 on IIT-AFF and UMD, against 72.84, 72.73 and 78.73 for our judge. Our loss to the retrained baselines on IIT-AFF cIoU comes from a few large contain and wrap grasp queries on which our judge accepts the whole vessel while the annotation marks only part of it; because cIoU pools pixels, these few queries decide the column. We tried purity targets, a frozen backbone, evidence-ranked gating, two- and three-judge agreement and area-weighted supervision. None closed the gap: the closest, requiring two judges to agree, raised that column from 64.4 to 68.3 under original words for an earlier variant of the judge, at the cost of two other columns.
What the alignment does and does not explain. The alignment changes only the words in Table 5. On UMD, the three operations it leaves untouched — cut, scoop and pound — score identically under both protocols, so the protocol cannot manufacture an advantage there; all of the change is in the four conflicting classes, and grasp alone moves from 15.7 to 70.3 gIoU. The alignment is therefore not a general boost but a correction on the classes whose annotation convention differs, and it is applied to every method it is reported for. It also does not hide our failures: on pound the retrained JevAny stays 16.2 points ahead of us under either protocol, largely through object-level mistakes that our gate does not revisit, which is the next paragraph’s subject rather than a vocabulary issue.
Confidence thresholds are brittle under a sharper judge. The gate opens only when both and fall below 0.9, and a sharp judge rarely lets them. On tenderizer_01/00000265/pound our judge gives the first extent candidate 0.998, so the gate stays closed and the prediction, entirely background, scores IoU 0; the retrained JevAny is less sure (0.536), backtracks to another hypothesis and reaches 0.88. For the judge trained without purity targets, replacing the absolute threshold by a comparison of part evidence across hypotheses raises UMD cIoU from 57.0 to 72.4 under original words, at a cost of 2.5 cIoU in domain. We keep the published rule in all tables; a control rule calibrated for one judge does not transfer to a sharper one.
What the readout contributes. The answer-embedding readout is measurable before any training. On the held-out photos, where the zero-initialised set block and geometry embedding still contribute nothing, the untrained model separates part candidates at AUROC 0.618, sub-parts at 0.569, and picks the correct object extent in 61.0% of groups against 33.3% chance; two epochs of training take these to 0.978, 0.956 and 92.7%. The region-token ablation also disables the readout, so it does not isolate the contribution of the answer embeddings. Scope of the aligned protocol. Both UMD’s handle and InstructPart’s body are defensible readings of grasp. The fixed mapping is a human choice, and its validity cannot be established on the same sets without circularity. We therefore report the original words too: our judge leads in domain, while retrained JevAny leads on UMD. The per-operation check above establishes only internal consistency: unchanged operations have identical predictions.
Cost. Judging a level in one pass reduces the judge to 8.1 requests and 8.1 encoded images per query, from 33.0 and 99.1 for a per-candidate interface, measured over the candidates actually visited on the 346 test queries. By contrast, Affordance-R1 writes on average 88 words of reasoning per query before it answers, while our judge generates no text. We do not report end-to-end latency: the segmenter stages are shared by all judges in Tab. 1 and our measurement covers the judge only.
Conclusion
We ground affordances with a System One model that confirms candidate regions instead of reasoning in text. The framework asks a hierarchy of region questions — object, part, sub-part — and passes evidence between the levels so that they correct one another; the model is trained for these questions from raw weights on the framework’s own requests and judges each level in one pass. Under definition-aligned evaluation it outperforms both System Two methods on every column and the System One baselines on ten of twelve, with four times fewer judge requests; the comparison with retrained baselines matches optimisation settings and epochs, while our supervision includes additional tasks. Large-container over-acceptance still costs IIT-AFF cIoU, released Valen leads on UMD , and an absolute confidence gate rarely opens under a sharp judge. Learning when to abstain and comparing evidence across object hypotheses remain open.
References
- [1]Tommaso Apicella, Alessio Xompero, Edoardo Ragusa, Riccardo Berta, Andrea Cavallaro, and Paolo Gastaldo. Affordance segmentation of hand-occluded containers from exocentric images. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, 2023. 2DOI
- [2]Tommaso Apicella, Alessio Xompero, Edoardo Ragusa, Riccardo Berta, Andrea Cavallaro, and Paolo Gastaldo. Real testing sets for visual affordance segmentation of hand-occluded objects. Zenodo, version 1.0.0, 2023.DOI
- [3]Tommaso Apicella, Alessio Xompero, Paolo Gastaldo, and Andrea Cavallaro. Segmenting object affordances: Reproducibility and sensitivity to scale. In European Conference on Computer Vision Workshops, 2024. 2DOI
- [4]Base Labs, Baseten, elephant-1: Official multimodal decision model. Hugging Face model release, 2026. Accessed 2026-10-25. 3
- [5]Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020. 3arxiv.org/abs/2005.14165
- [6]Zhaowei Cai and Nuno Vasconcelos. Cascade R-CNN: Delving into high quality object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018. 2arxiv.org/abs/1712.00726
- [7]Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Proceedings of the European Conference on Computer Vision, 2020. 3arxiv.org/abs/2005.12872
- [8]Nicolas Carion et al. SAM 3: Segment anything with concepts. Meta AI model release, 2025. 2, 3arxiv.org/abs/2511.16719
- [9]DeepSeek-AI. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. 3
- [10]Thanh-Toan Do, Anh Nguyen, and Ian Reid. AffordanceNet: An end-to-end deep learning approach for object affordance detection. In IEEE International Conference on Robotics and Automation, 2018. 2
- [11]James J. Gibson. The Ecological Approach to Visual Perception. Houghton Mifflin, 1979. 2
- [12]H2O.ai. H2O-Lightning-4B: Model card and inference implementation. Hugging Face model release, 2026. Version 1.4.0. Accessed 2026-10-25. 3, 4, 5, 6
- [13]Ju He, Shuo Yang, Shaokang Yang, Adam Kortylewski, Xiaoding Yuan, Jie-Neng Chen, Shuai Liu, Cheng Yang, Qihang Yu, and Alan Yuille. PartImageNet: A large, high-quality dataset of parts. In European Conference on Computer Vision, 2022. 2arxiv.org/abs/2112.00933
- [14]Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE International Conference on Computer Vision, 2017. 3
- [15]Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank adaptation of large language models. In International Conference on Learning Representations, 2022. 5arxiv.org/abs/2106.09685
- [16]Dengyang Jiang, Zanyi Wang, Hengzhuang Li, Sizhe Dang, Teli Ma, Wei Wei, Guang Dai, Lei Zhang, Harry Yang, and Mengmeng Wang. AffordanceSAM: Segment anything once more in affordance grounding. arXiv preprint arXiv:2504.15650, 2025. 2arxiv.org/abs/2504.15650
- [17]Daniel Kahneman. Thinking, Fast and Slow. Farrar, Straus and Giroux, 2011. 1
- [18]Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023. 2, 3arxiv.org/abs/2304.02643
- [19]Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shiyu Liu, and Jiaya Jia. LISA: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9579–9589, 2024. 2arxiv.org/abs/2308.00692
- [20]Gen Li, Varun Jampani, Deqing Sun, and Laura Sevilla-Lara. LOCATE: Localize and transfer object parts for weakly supervised affordance grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. 2arxiv.org/abs/2303.09665
- [21]Gen Li, Deqing Sun, Laura Sevilla-Lara, and Varun Jampani. One-shot open affordance learning with foundation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. 2arxiv.org/abs/2311.17776
- [22]Junyi Li, Junfeng Wu, Weizhi Zhao, Song Bai, and Xiang Bai. PartGLEE: A foundation model for recognizing and parsing any objects. In European Conference on Computer Vision, 2024. 2, 3arxiv.org/abs/2407.16696
- [23]Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In International Conference on Learning Representations, 2024. 3arxiv.org/abs/2305.20050
- [24]Xinrui Lin, Sha Zhang, Shumin Wang, Zhenghuan Zhu, Jiajun Deng, and Yanyong Zhang. ThinkinAfford: Affordance-centric reasoning for fine-grained 3D grounding in cluttered scenes. arXiv preprint arXiv:2608.10981, 2026. 2arxiv.org/abs/2608.10981
- [25]Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qiang Jiang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection. In European Conference on Computer Vision, 2024. 2arxiv.org/abs/2303.05499
- [26]Yuqi Liu, Tianyuan Qu, Zhisheng Zhong, Bohao Peng, Shu Liu, Bei Yu, and JiaJia Jia. VisionReasoner: Unified reasoning-integrated visual perception via reinforcement learning. In International Conference on Learning Representations, 2026. 1, 2, 5, 6arxiv.org/abs/2505.12081
- [27]Hongchen Luo, Wei Zhai, Jing Zhang, Yang Cao, and Dacheng Tao. Learning affordance grounding from exocentric images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. 2arxiv.org/abs/2203.09905
- [28]Kaichun Mo, Leonidas J. Guibas, Mustafa Mukadam, Abhinav Gupta, and Shubham Tulsiani. Where2Act: From pixels to actions for articulated 3D objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021. 2
- [29]Austin Myers, Ching L. Teo, Cornelia Fermüller, and Yiannis Aloimonos. Affordance detection of tool parts from geometric features. In IEEE International Conference on Robotics and Automation, 2015. 1, 2, 5DOI
- [30]Tushar Nagarajan, Christoph Feichtenhofer, and Kristen Grauman. Grounded human-object interaction hotspots from video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019. 2arxiv.org/abs/1812.04558
- [31]Anh Nguyen, Dimitrios Kanoulas, Darwin G. Caldwell, and Nikos Tsagarakis. Object-based affordances detection with convolutional neural networks and dense conditional random fields. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2017. 2, 5
- [32]Shengyi Qian, Weifeng Chen, Min Bai, Xiong Zhou, Zhuowen Tu, and Li Erran Li. AffordanceLLM: Grounding affordance from vision language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 7587–7597, 2024. 2arxiv.org/abs/2401.06341
- [33]Qwen Team. Qwen3.5-4B: Official model release. Hugging Face model card, 2026. Checkpoint revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. 2, 4
- [34]Vignesh Ramanathan et al. PACO: Parts and attributes of common objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. 2arxiv.org/abs/2301.01795
- [35]Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. SAM 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024. 2arxiv.org/abs/2408.00714
- [36]Timo Schick and Hinrich Schütze. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, 2021. 3arxiv.org/abs/2001.07676
- [37]SimpleJev. JevAny-Qwen3.5-4B-LoRA: Official pointer-readout checkpoint. Hugging Face model release, 2026. Official runtime v0.3.0. Accessed 2026-10-25. 3, 4, 5, 6
- [38]Peize Sun, Shoufa Chen, Chenchen Zhu, Fanyi Xiao, Ping Luo, Saining Xie, and Zhicheng Yan. Going denser with open-vocabulary part segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023. 2arxiv.org/abs/2305.11173
- [39]TypeSafe. Jev 1.14: Hosted system one model and api documentation. Official model and API documentation, 2026. Pinned API version jev-1.14.0. Accessed 2026-10-26. 3
- [40]Valen Team. Valen-4B: Model card and inference implementation. Hugging Face model release, 2026. Accessed October 22, 2026. 1, 3, 4, 5, 6
- [41]Zifu Wan, Yaqi Xie, Ce Zhang, Zhiqiu Lin, Zihan Wang, Simon Stepputtis, Deva Ramanan, and Katia Sycara. Instruct-Part: Task-oriented part segmentation with instruction reasoning. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2025. 2, 5
- [42]Hanqing Wang, Shaoyang Wang, Yiming Zhong, Zemin Yang, Jiamin Wang, Zhiqing Cui, Jiahao Yuan, Yifan Han, Mingyu Liu, and Yuexin Ma. Affordance-RL: Reinforcement learning for generalizable affordance reasoning in multimodal large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 9738–9746, 2026. 1, 2, 5, 6
- [43]Zhaoning Wang, Yi Zhong, Jiawei Fu, Henrik I. Christensen, and Jun Gao. AFUN: Towards an affordance foundation model for functionality understanding. arXiv preprint arXiv:2606.02551, 2026. 2arxiv.org/abs/2606.02551
- [44]Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022. 3arxiv.org/abs/2201.11903
- [45]Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in GPT-4V. arXiv preprint arXiv:2310.11441, 2023. 2arxiv.org/abs/2310.11441
- [46]Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. In International Conference on Learning Representations, 2024. 3arxiv.org/abs/2310.07704
- [47]Qing Zhang, Xuesong Li, and Jing Zhang. Probing and bridging geometry-interaction cues for affordance reasoning in vision foundation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2026. 2arxiv.org/abs/2602.20501
- [48]Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Yu Liu, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on region-of-interest. arXiv preprint arXiv:2307.03601, 2023. 3arxiv.org/abs/2307.03601
- [49]Zixin Zhang, Kanghao Chen, Hanqing Wang, Hongfei Zhang, Harold Haodong Chen, Chenfei Liao, Litao Guo, Yinchuan Li, and Ying-Cong Chen. A4-Agent: An agentic framework for zero-shot affordance reasoning. In European Conference on Computer Vision, 2026. 2arxiv.org/abs/2512.14442
- [50]Liamin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems, Datasets and Benchmarks Track, 2023. 3arxiv.org/abs/2306.05685
