betaFeedback

WorksComputer Science

SEER: Semantic Evidence for Executable Region Propagation in Open-Vocabulary Semantic Segmentation

Abstract

Training-free open-vocabulary semantic segmentation (OVSS) has made strong progress by adapting CLIP-like representations to dense prediction, but its remaining errors are often local, structured, and tied to the active label space. A successful correction must be visually plausible, legal under the target vocabulary, and executable in the segmentation score field. We propose SEER, a training-free test-time repair layer for Semantic Evidence and Executable Region propagation. SEER casts post-prediction OVSS repair as evidence-conditioned label propagation. The OVSS score space proposes compact candidate labels, a frozen VLM verifies closed-set regional evidence, and an execution gate certifies whether the evidence is legally and semantically actionable. The accepted label is then propagated by an OVSS-side policy: coherent high-confidence regions allow broad propagation, while object- or boundary-risk cases use conservative score-grounded propagation. Across eight standard OVSS benchmarks, SEER reaches 58.0 average mIoU without updating OVSS weights. Matched-input controls isolate the repair mechanism, while resource measurements report the cost of the complete frozen system.

Introduction

Open-vocabulary semantic segmentation (OVSS) assigns dense labels from an open or user-specified vocabulary. Starting from early open-vocabulary scene parsing Zhao et al. [51], the field has been accelerated by CLIP Radford et al. [32] and OpenCLIP Cherti et al. [12]. Training-free methods further improve dense prediction by modifying CLIP attention, calibrating patch-text similarity, and injecting visual priors. MaskCLIP Zhou et al. [54] exposes dense CLIP predictions, SCLIP Wang et al. [40] adapts self-attention for segmentation, ProxyCLIP Lan et al. [24] introduces proxy features, Trident Shi et al. [37] restructures feature aggregation, and CorrCLIP Zhang et al. [50] corrects patch correlation. As these systems become stronger, the bottleneck shifts from global recognition to local residues such as boundary spillovers, object-part confusions, and stuff/object ambiguities.

Residual repair requires more than crop classification. OVSS predictions live under an active label space, where a visually meaningful region can be non-target when its label is absent, while a scene or stuff region becomes executable once the corresponding label enters the vocabulary. A repair action therefore has three coupled requirements. The label must be visually supported, legal under the active vocabulary, and propagated to the right pixels. Regional evidence can suggest a semantic direction, while pixel support must still be recovered from the dense prediction.

General-purpose vision-language models (VLMs) provide a natural source of regional semantic evidence. Qwen-VL Qwen Team [31], LLaVA-OneVision-1.5 An et al. [1], and GPT-4o Hurst et al. [18] can compare candidate descriptions and answer localized visual questions. Direct pixel decisions remain unreliable because regional semantics, label-space validity, and pixel execution become entangled in one response. A VLM may name the highlighted region correctly without knowing whether the label is executable in the current OVSS score field or how far it should propagate inside the proposal. We therefore restrict the VLM to regional evidence and retain pixel authority in the segmentation model.

This paper studies how regional VLM evidence can be made executable in OVSS label propagation. Our answer is SEER, short for Semantic Evidence and Executable Region propagation. SEER uses prediction-side signals from the baseline mask and score tensor to select auditable proposals, asks a VLM to verify a compact score-derived candidate set with uncertain as an abstention option, and accepts evidence only if it passes executable gates. After certification, the OVSS score field executes the decision through a propagation policy. Coherent high-confidence regions can use broad label propagation, while object-like or boundary-sensitive regions use conservative support derived from the underlying OVSS score field for pixel-level propagation.

This framing keeps the method tied to OVSS. Labels are proposed by open-vocabulary scores, action validity is checked against the active label space, and pixel changes are produced by score-space propagation. Broad propagation is used after closed-set evidence verification, while conservative propagation remains the fallback when boundaries or selector noise make broad support risky. Across eight standard benchmarks, SEER reaches 58.0 average mIoU with the ViT-L/14 base. Fixed-prediction controls hold the score tensor, proposals, candidates, parser, and audit budget constant while varying the repair signal or executor. We also report complete-system latency and memory, lighter verifiers, proposal-source transfer, sensitivity tests, and residual failures.

This paper makes three contributions.

  • We formulate OVSS post-prediction repair as executable region propagation, separating regional semantic evidence, label-space validity, and pixel support.

  • SEER is a training-free evidence-to-execution layer that uses frozen VLM verification while leaving pixel propagation to the OVSS score field.

  • Systematic diagnostics evaluate compact candidates, executable gates, multi-view evidence, evidence-conditioned propagation, and complete-system cost.

Related Work

Open-vocabulary semantic segmentation. OVSS extends zero-shot and open-vocabulary recognition to dense prediction Zhao et al. [51]; Bucher et al. [5]; Xian et al. [44]. CLIP Radford et al. [32] and OpenCLIP Cherti et al. [12] provide the basic image-text alignment, but their image-level training objective does not directly produce spatially coherent patch predictions. Training-free OVSS methods therefore change how CLIP features are extracted, calibrated, or aggregated. MaskCLIP Zhou et al. [54] exposes dense CLIP predictions, CLIP Surgery Li et al. (2025) modifies

CLIP behavior for open-vocabulary localization, SCLIP Wang et al. [40] rethinks self-attention for dense inference, CLIPtrase Shao et al. [36] studies CLIP’s segmentation ability, and ProxyCLIP Lan et al. [24] uses proxy attention from stronger visual features. More recent methods improve calibration, residual structure, correlations, and feature aggregation Lan et al. [23]; Hajimiri et al. [17]; Yang et al. [49]; Bai et al. [2]; Sun et al. [38]; Kim et al. [20]; Wang et al. [41]; Shi et al. [37]; Zhang et al. [50]. These works mainly seek a stronger initial OVSS prediction. SEER instead repairs residual local errors after that prediction is fixed.

Region priors and label propagation. Region and structure priors are widely used in segmentation. Training-based open-vocabulary systems introduce dense supervision, adapters, mask classification, or generative priors Rao et al. [33]; Xu et al. [46]; Liang et al. [27]; Xu et al. [48]; Cho et al. [13]; Lee et al. [25]; Zou et al. [56]; Zou et al. [57]; Xu et al. [47]; Cha et al. [8]; Wu et al. [42]. Class-agnostic and grounding models such as SAM, SAM2, EntitySeg, RegionCLIP, and Grounding DINO provide masks, regions, or localization priors that are useful beyond their original tasks Kirillov et al. [21]; Ravi et al. [34]; Qi et al. (2023); Zhong et al. [52]; Liu et al. [28]. Classical CRFs, graph cuts, propagation methods, and modern segmentation architectures also show that spatial consistency and pixel-level confidence matter for reliable masks Krähenbühl & Koltun [22]; Rother et al. [35]; Chen et al. [10]; Xie et al. [45]; Cheng et al. [11] (2022). SEER treats each proposal as a candidate domain over which verified evidence may propagate. The propagation distance is conditioned on evidence confidence, region form, score support, and risk.

Vision-language models for visual verification. Modern VLMs can answer region-level questions, compare candidate descriptions, and provide visual evidence that is not captured by local score margins Qwen Team [31]; An et al. [1]; Zhu et al. [55]; Clark et al. [14]; Hurst et al. [18]; Bousselham et al. [4]. This makes them appealing for segmentation repair, but free-form VLM outputs are poorly matched to a fixed OVSS label space. A generic region classifier may name the crop, yet a repair system must also know whether the label is legal and whether the segmentation system can execute the label at pixels. SEER uses the VLM only as a closed-set verifier over score-derived candidates. The executable decision remains inside the OVSS label and score space, which separates our setting from direct mask correction or ordinary region classification.

Method

Overview. SEER repairs a fixed OVSS prediction in four steps (Figure 2). It (1) selects a small proposal subset, (2) builds legal candidate labels from the frozen score tensor, (3) asks a frozen VLM to select one candidate or abstain, and (4) lets the OVSS scores decide which pixels may change. High-confidence coherent regions may use the full proposal; object-like or boundary-sensitive cases use a conservative score-supported subset. The VLM therefore supplies only a regional label decision and never predicts or edits a mask.

Overview of SEER

Figure 2. Overview of SEER. SEER converts regional VLM evidence into executable OVSS label propagation. From a fixed prediction, it selects ambiguous proposals, derives score-based candidates, and verifies them with multi-view closed-set reasoning. Score support and vocabulary legality certify the verified label, and the OVSS model controls its pixel propagation.

Problem setup: a label and its pixel support

Let I∈RH×W×3I \in\mathbb{R}^{H \times W \times3} be an image and Ω={1,…,H}×{1,…,W}\Omega= \{1,\ldots,H\} \times\{1,\ldots,W\} be the pixel lattice. The active OVSS label space is C={1,…,C}C = \{1,\ldots,C\}. A fixed OVSS model outputs a prediction map Y0∈CH×WY^0 \in C^{H \times W} and a dense score tensor S∈RH×W×CS \in\mathbb{R}^{H \times W \times C}. We use Sp,cS_{p,c} for the score of class cc at pixel pp, and mpm_p for the top-1/top-2 score margin. A fixed SAM2-based class-agnostic proposal bank provides regions M={Mi}i=1N\mathcal{M} = \{M_i\}_{i=1}^{N}, where Mi⊆ΩM_i \subseteq\Omega. The proposal bank is not a semantic predictor in our method; it only defines candidate spatial domains for later evidence verification and score-side execution.

For a proposal MiM_i, a regional label is not yet a repair. It becomes executable only after the system determines how the label should propagate. We write an executable propagation action as

ri=(ℓi,Wi),ℓi∈C,Wi⊆Mi,(1)r_i = (\ell_i, W_i), \qquad\ell_i \in C,\quad W_i \subseteq M_i, \tag*{(1)}

where ℓi\ell_i is the verified target label and WiW_i is the propagation support. Applying rir_i updates the current prediction by

Ypt+1={ℓi,p∈Wi,Ypt,p∉Wi,(2)Y_p^{t+1} = \begin{cases} \ell_i, & p \in W_i,\\ Y_p^t, & p \notin W_i, \end{cases} \tag*{(2)}

where tt denotes the execution order. Regional evidence determines the target label, and the OVSS system determines the propagation support WiW_i.

For clarity, we write the three stages of SEER as

M→PrioritizeR(I),(I,Mi,Ai)→Verifyai,(S,Y0,Mi,ai)→Propagate(ℓi,Wi).(3)\mathcal{M} \xrightarrow{\text{Prioritize}} \mathcal{R}(I), \qquad (I,M_i,A_i) \xrightarrow{\text{Verify}} a_i, \qquad (S,Y^0,M_i,a_i) \xrightarrow{\text{Propagate}} (\ell_i,W_i). \tag*{(3)}

Here R(I)⊆M\mathcal{R}(I) \subseteq\mathcal{M} is the selected auditable subset, AiA_i is the candidate label set constructed below, and aia_i is the VLM evidence tuple. This factorization is the design principle: region selection limits the VLM budget, verification extracts closed-set evidence, and propagation decides whether the evidence should spread broadly or conservatively.

The pair (ℓi,Wi)(\ell_i,W_i) makes the division of labor explicit: the VLM helps choose a regional label, while the OVSS scores choose the pixels that can receive it. A correct label can still increase error if WiW_i crosses an already-correct boundary, and a safe support can miss useful corrections if it is too small. Appendix B gives the exact benefit–harm decomposition and a short non-identifiability result showing why regional evidence alone cannot determine pixel support.

Choosing proposals under an audit budget

Querying every proposal is unnecessary and expensive. SEER selects a budgeted subset using information available before VLM inference. For each proposal, the dominant baseline label is

bi=mode⁡{Yp0∣p∈Mi}.(4)b_i = \operatorname{mode}\{Y^0_p \mid p \in M_i\}. \tag*{(4)}

The mean score of class cc inside the proposal is

sˉi,c=1∣Mi∣∑p∈MiSp,c(5)\bar{s}_{i,c} = \frac{1}{|M_i|}\sum_{p \in M_i} S_{p,c} \tag*{(5)}

and the strongest non-baseline competitor is

ci=arg⁡max⁡c∈C∖{bi}sˉi,c.(6)c_i = \arg\max_{c \in C \setminus\{b_i\}} \bar{s}_{i,c}. \tag*{(6)}

The priority function is a fixed utility proxy over four prediction-side signals: regional coherence, potential impact, score-space competition, and risk exposure. We write

πi=π(coh⁡i,area⁡i,comp⁡i,risk⁡i),(7)\pi_i = \pi(\operatorname{coh}_i,\operatorname{area}_i,\operatorname{comp}_i,\operatorname{risk}_i), \tag*{(7)}

where coh⁡i\operatorname{coh}_i measures the consistency of Y0Y^0 inside MiM_i, area⁡i=∣Mi∣/∣Ω∣\operatorname{area}_i = |M_i|/|\Omega|, comp⁡i\operatorname{comp}_i measures the score competition between cic_i and bib_i, and risk⁡i\operatorname{risk}_i measures exposure to object or boundary-sensitive predictions. All terms are computed from Y0Y^0, SS, and MiM_i. The exact constants and routing predicates are fixed once and listed in Appendix C. The priority score is a budget controller that exposes a small auditable set. The selected set under budget KK is

R(I)=TopK⁡Mi∈Mπi.(8)\mathcal{R}(I) = \operatorname{TopK}_{M_i \in\mathcal{M}} \pi_i. \tag*{(8)}

Closed-set regional verification

For each selected proposal Mi∈R(I)M_i \in\mathcal{R}(I), SEER constructs a compact candidate set in the OVSS label space:

Ai={bi}∪Qi∪{uncertain},(9)\mathcal{A}_i = \{b_i\} \cup Q_i \cup\{\text{uncertain}\}, \tag*{(9)}

where Qi⊆C∖{bi}Q_i \subseteq C \setminus\{b_i\} contains up to three score-derived alternatives from proposal-averaged scores and local score support. In the default setting, this gives a closed candidate set consisting of the current base label, the top-3 non-base alternatives, and uncertain. The VLM is not asked to search the full vocabulary or invent a label; labels outside Ai\mathcal{A}_i are not executable.

The VLM sees multiple views of the same proposal, including the full image with MiM_i highlighted and cropped views around the proposal. It returns an evidence tuple

ai=(ℓ^i,conf⁡i,form⁡i),(10)a_i = (\hat{\ell}_i, \operatorname{conf}_i, \operatorname{form}_i), \tag*{(10)}

where ℓ^i∈Ai\hat{\ell}_i \in\mathcal{A}_i is the selected label, conf⁡i∈{high,medium,low}\operatorname{conf}_i \in\{\text{high}, \text{medium}, \text{low}\} is confidence, and form⁡i\operatorname{form}_i indicates whether the region is coherent, boundary-like, mixed, or unclear. This tuple specifies a label direction and a region form; pixel support is still decided by the OVSS score field.

Filtering VLM answers before writeback

The VLM answer is filtered before it can affect any pixel. The executable indicator is

ei=1[ℓ^i∈C, ℓ^i≠bi, ℓ^i≠uncertain, conf⁡i=high, form⁡i∉{mixed,unclear}].(11)e_i = \mathbf{1}\left[\hat{\ell}_i \in C,\ \hat{\ell}_i \ne b_i,\ \hat{\ell}_i \ne\text{uncertain},\ \operatorname{conf}_i = \text{high},\ \operatorname{form}_i \notin\{\text{mixed}, \text{unclear}\}\right]. \tag*{(11)}

If ei=0e_i = 0, the proposal leaves the prediction unchanged. If ei=1e_i = 1, the target label is ℓi=ℓ^i\ell_i = \hat{\ell}_i. Thus every accepted answer must be legal, high-confidence, different from the current label, and not mixed or unclear. Boundary-like answers may still be used, but only through conservative pixel support. § 4.4 and Appendix § 14 test the filter and propagation choices under matched inputs.

Score-controlled pixel updates

Once the label is certified, the remaining question is how far the evidence should propagate. SEER uses a policy family Γ\Gamma that maps certified evidence and OVSS scores to a support set:

Wi=Γ(S,Y0,Mi,ℓi,ai),Wi⊆Mi.(12)W_i = \Gamma(S, Y^0, M_i, \ell_i, a_i), \qquad W_i \subseteq M_i. \tag*{(12)}

We use three propagation regimes. Coherent propagation sets Wi=MiW_i = M_i when a region is coherent, high-confidence, and low-risk; in this case, the proposal is treated as a semantic unit. Score-grounded propagation propagates only to pixels where the target remains supported by the OVSS score field. Boundary-conservative propagation restricts propagation to low-margin pixels for object-like or risk-sensitive cases.

For score-grounded propagation, let

yptop=arg⁡max⁡c∈C Sp,c.(13)y_p^{\mathrm{top}} = \underset{c \in C}{\arg\max}\ S_{p,c}. \tag*{(13)}

We use the support predicate

ρ(p,ℓi,bi)=1[yptop=ℓi∨Sp,ℓi≥Sp,bi−τs∨(mp≤τm∧Sp,ℓi≥Sp,bi−τl)],(14)\rho(p, \ell_i, b_i) = \mathbf{1}\left[y_p^{\mathrm{top}} = \ell_i \lor S_{p,\ell_i} \ge S_{p,b_i} - \tau_s \lor\left(m_p \le\tau_m \land S_{p,\ell_i} \ge S_{p,b_i} - \tau_l\right)\right], \tag*{(14)}

where τs,τm,τl\tau_s, \tau_m, \tau_l are fixed thresholds. The score-grounded support is Wi={p∈Mi∣ei=1,ρ(p,ℓi,bi)=1}W_i = \{p \in M_i \mid e_i = 1, \rho(p, \ell_i, b_i) = 1\}; the boundary-conservative support further restricts it to low-margin pixels. In practice, SEER uses the same deterministic contract for all datasets: broad propagation is reserved for reliable context/stuff evidence, while object, small-region, boundary, or high-risk cases use conservative support after closed-set certification.

Experiments

Experimental Setup

Evaluation protocol. Following common open-vocabulary semantic segmentation (OVSS) protocols Zhou et al. [54]; Wang et al. [40]; Lan et al. [24]; Shi et al. [37]; Zhang et al. [50], we evaluate on eight benchmarks from six validation sets: PASCAL VOC with and without background (VOC21/VOC20) Everingham et al. [16], PASCAL Context with and without background (PC60/PC59) Mottaghi et al. [29], COCO Object and COCO-Stuff164K Caesar et al. [6], ADE20K Zhou et al. [53], and Cityscapes Cordts et al. [15]. We report mean Intersection over Union (mIoU). Comparisons with prior methods follow the CorrCLIP-style evaluation configuration: published numbers are grouped by CLIP backbone size, and our method is evaluated with both CLIP ViT-B/16 and ViT-L/14 settings.

Implementation details. We follow the CorrCLIP evaluation configuration, while the repair input itself is a fixed CLIP score field SS and its induced prediction Y0Y^{0}. Region domains come from a fixed SAM2 Ravi et al., [34] proposal bank and are never treated as semantic predictions. The default verifier is the locally deployable Qwen3.5-9B [31], chosen for strong regional recognition, open weights, and deterministic decoding; it is not fine-tuned on any evaluation benchmark. Its public documentation does not provide a sample-level pretraining manifest, so we do not assume zero overlap with segmentation or grounding data and instead test replacement by three other VLM families and CLIP. The VLM receives proposal views and legal label names. Ground truth, target masks, and pixel-write supervision are excluded.

For mechanism studies, we fix Y0Y^{0}, SS, the proposal bank, candidates, parser, gate, propagation policy, audit inputs, and evaluation split; only the tested repair signal or executor changes. This fixed-prediction protocol uses MetaCLIP FullCC ViT-B/16 QuickGELU, frozen DINO ViT-B/8, benchmark-native mappings, and aspect-ratio-preserving resize (2048, 336) for VOC/PC (2048, 448) for City/ADE. Its initial scores are a separate locally frozen diagnostic state, not a reproduction of the published CorrCLIP row in Table 1. The constants were selected once from aggregate diagnostics on the evaluated natural-image validation domains, then frozen across all eight benchmarks, two backbones, four OVSS bases, and every verifier; we do not claim domain-blind development or medical/remote-sensing validation. Appendix § 10 lists the complete contract and sensitivity tests.

MethodSizeVOC21VOC20PC60PC59ObjectStuffADECityAvg.
Training-based
TCL Cha et al. [8]ViT-B/1655.083.230.433.931.622.417.124.037.2
CLIP-DINOiser Wysoczanska et al. [43]ViT-B/1662.180.932.435.934.824.620.031.740.3
CoDe Wu et al. [42]ViT-B/1657.7–30.5–32.323.917.728.9–
LoGoSeg Chen et al. [9]ViT-B/1678.695.4–58.2––32.6––
ESC-Net Lee et al. [25]ViT-B/1680.197.3–59.0––35.6––
Training-free
CLIP Radford et al. [32]ViT-B/1611.541.94.49.21.64.42.95.010.1
MaskCLIP Zhou et al. [54]ViT-B/1638.874.923.626.420.616.49.812.627.9
ClearCLIP Lan et al. [23]ViT-B/1651.880.932.635.933.023.916.730.038.1
SCLIP Wang et al. [40]ViT-B/1659.180.430.434.230.522.416.132.238.2
ProxyCLIP Lan et al. [24]ViT-B/1661.380.335.339.137.526.520.238.142.3
LaVG Kang & Cho [19]ViT-B/1662.182.531.634.734.223.215.826.238.8
CLIPtrase Shao et al. [36]ViT-B/1653.081.230.834.944.824.117.0––
NACLIP Hajimiri et al. [17]ViT-B/1664.183.035.038.436.225.719.138.342.5
Trident Shi et al. [37]ViT-B/1667.184.538.642.241.128.321.942.945.8
ResCLIP Yang et al. [49]ViT-B/1661.186.033.536.835.024.718.035.941.4
SC-CLIP Bai et al. [2]ViT-B/1664.684.336.840.137.726.620.141.043.9
CLIPer Sun et al. [38]ViT-B/1665.985.237.641.739.027.521.4––
CASS Kim et al. [20]ViT-B/1665.887.836.740.237.826.720.439.444.4
CorrCLIP Zhang et al. [50]ViT-B/1674.888.844.248.843.731.626.949.451.0
SEER (Ours)ViT-B/1679.3 (+4.5)93.8 (+5.0)47.7 (+3.5)53.9 (+5.1)48.5 (+3.7)36.2 (+4.6)35.2 (+8.3)55.0 (+5.6)56.2 (+5.2)
FreeDA Barsellotti et al. [3]ViT-L/1455.487.938.343.537.428.823.236.743.9
CaR Sun et al. [39]ViT-L/1467.691.430.539.536.6–17.7––
ProxyCLIP Lan et al. [24]ViT-L/1460.683.234.537.739.225.622.640.143.0
ResCLIP Yang et al. [49]ViT-L/1454.185.530.934.532.523.418.233.739.1
SC-CLIP Bai et al. [2]ViT-L/1465.088.336.940.640.526.921.741.345.2
CLIPer Sun et al. [38]ViT-L/1469.890.038.043.643.328.724.4––
CorrCLIP Zhang et al. [50]ViT-L/1476.791.544.950.849.434.030.751.153.6
SEER (Ours)ViT-L/1479.6 (+2.9)94.2 (+2.7)48.2 (+3.3)55.2 (+4.4)52.9 (+3.5)38.6 (+4.6)38.3 (+7.6)56.9 (+5.8)58.0 (+4.4)

Table 1. Complete-system accuracy on eight OVSS benchmarks. Published baseline numbers follow the CorrCLIP-style evaluation configuration and are grouped by CLIP size. SEER is an accuracy–cost point, not an equal-resource comparison: it adds fixed SAM2 proposals and sparse local Qwen3.5-9B verification to CorrCLIP. Paired end-to-end cost and matched-input attribution are reported in Table 7 and Table 4.

ConfigurationProposal gen.OVSSAuditWritebackTotalOverhead
Frozen OVSS—5.042——5.042—
Cached SEER—5.0420.4240.0435.509+9.3%
Cold SEER0.5085.0420.4240.0436.018+19.3%

Table 7. End-to-end cost ($/image). Validation-weighted measurements on the same hardware. Cached SEER reuses fixed SAM2 proposals; cold SEER includes proposal generation. Audit load averages 0.71 calls per image.

Comparison with State-of-the-Art Methods

Complete-system comparison. Table 1 compares complete-system accuracy using published baseline results; it does not imply matched hardware or equal external resources. With both ViT-B/16 and ViT-L/14, SEER improves its corresponding CorrCLIP base on every benchmark, with the largest gains on ADE20K and Cityscapes. The remaining experiments use one frozen prediction state and matched proposal/audit inputs to attribute the gain, while Tab. 7 reports the paired cost on our hardware.

Fixed-Prediction Repair Analysis

Fixed-prediction repair. The fixed-prediction diagnostic in Figure 1 reports the controlled setting used in the mechanism analysis. The prediction map and score tensor are fixed, and only the evidence-to-execution layer is applied. Gains across eight benchmarks rule out changes in backbone, resolution, or training recipe, supporting the premise that residual OVSS errors are repairable once regional evidence becomes executable.

SEER turns regional semantic evidence into executable OVSS label propagation

Figure 1. SEER turns regional semantic evidence into executable OVSS label propagation. Starting from a fixed OVSS prediction, SEER selects ambiguous proposals, derives compact candidate labels from the active score space, and asks a VLM to verify regional evidence. The OVSS system executes certified evidence: score-supported seeds guide label propagation, while boundary-conservative guards restrict unsafe spreading near object and structure boundaries. The bar chart shows consistent mIoU gains across diverse OVSS benchmarks.

Plug-in transfer across OVSS bases. We next ask whether the repair layer is tied to one score geometry. The same SEER pipeline is applied to ProxyCLIP, Trident, ResCLIP, and CorrCLIP: proposals, candidates, VLM auditing, filters, propagation rules, coefficients, and thresholds stay fixed across bases and benchmarks. Only the OVSS prediction and score cache change. Every reported base–benchmark pair improves, with mean gains from 4.53 to 5.93 mIoU; full results and confidence intervals appear in Appendix I.

Mechanism Analysis

Component chain. The next experiments follow the method decomposition. The full-chain diagnostic in Table 2 compares SEER with simpler repair configurations. Each alternative breaks one link in the deterministic execution contract: visual verification, label-space restriction, multi-view context, or high-recall coherent propagation. Within this diagnostic, all variants reduce the average gain, supporting SEER as a coupled design whose gain depends on both regional evidence and score-side execution.

SettingVLMClosedMulti-viewCoherentVOC21PC60CityADEAvg.
Initial prediction72.8844.0147.4128.6348.23
Score-only rule✓+2.10+3.80+2.53+2.13+2.64
Free-form VLM✓✓✓+2.82+0.00+0.00+0.00+0.70
Single view✓✓✓+2.43+0.30+1.68+0.67+1.27
Conservative exec.✓✓✓✓+3.73+2.47+4.05+7.06+4.33
SEER✓✓✓✓+6.72+4.19+9.49+9.67+7.52

Table 2. Component diagnostics. Each row is an alternative repair setting, not a cumulative build-up. The initial values are an independently frozen diagnostic SS, Y0Y^{0}, not the published CorrCLIP row in Tab. 1; every delta uses this identical state.

Evidence interface and execution. The interface study in Table 3 shows why the verifier should operate in a bounded label space. Free-form answers may miss the active vocabulary, while the full vocabulary introduces legal but weakly supported choices. With the default top-3 alternatives, the analysis-only target-in-set coverage is 99.9% across the four mechanism benchmarks. Filtering raises decision precision from 57.9% before execution to 82.3% among executed answers; 36.2% of audits leave the prediction unchanged and only 0.2% are invalid or out of set (Appendix D). Figure 3 gives the matching pixel-support story: strict routing is safe but low-recall, whereas SEER expands only high-confidence, score-compatible regions.

InterfaceVOC21PC60CityADEAvg.
Free-form+2.82+0.00+0.00+0.00+0.70
Full label list+5.97+0.55+6.02+6.56+4.78
Closed top-1+5.68+1.38+7.95+8.43+5.86
Closed top-2+6.94+2.22+6.79+8.93+6.22
Closed top-3+6.72+4.19+9.49+9.67+7.52

Table 3. Evidence interface. Free-form asks the VLM to name a label, full label list exposes the whole vocabulary, and closed top-kk contains the current/base label, up to kk non-base score-derived alternatives, and uncertain.

Execution frontier bar chart comparing VOC, PC, ADE, and support across Strict, Lenient, and SEER settings

Figure 3. Execution frontier. Bars show Δ\Delta mIoU on VOC, PC, and ADE; the dashed line reports certified support. SEER reaches the strongest gains by turning broader verified evidence into executable propagation.

Controls and visual context. Under the matched inputs in Table 4, a score rule, ordinary CLIP crop classifier, VLM answer with only score support, and Dense-CRF smoothing all remain below full SEER. The gain therefore is not reproduced by proposals plus a heuristic, a stronger regional classifier, an isolated VLM answer, or generic smoothing alone. The view ablation in Table 5 further shows that a tight crop is insufficient: global scene context, local boundaries, and region appearance provide complementary evidence.

ControlVOC21PC60CityADEAvg.
OVSS score rule+2.10+3.80+2.53+2.13+2.64
CLIP crop classifier+3.88+2.38+2.70+5.80+3.69
VLM + score support+2.09+0.87+1.74+2.11+1.70
Dense-CRF smoothing+0.01−0.01+0.02+0.01+0.01
SEER+6.72+4.19+9.49+9.67+7.52

Table 4. Matched-input controls. Every row uses the same frozen prediction, proposals, candidates, audit pool, and 2,000-audit cap.

ViewVOC21PC60CityADEAvg.
Full image+4.26+2.18+4.43+4.71+3.90
Local crop+4.91+2.76+5.88+4.64+4.55
Tight crop+3.48+2.07+3.96+3.52+3.26
Three views+6.72+4.19+9.49+9.67+7.52

Table 5. Multi-view evidence. Results use the default audit pool, verifier, candidates, gate, and propagation policy.

FamilyScaleVOC21PC60CityADEAvg.
Qwen0.8B+2.23+0.34+0.70+2.64+1.48
Qwen4B+5.86+3.26+7.49+8.45+6.26
Qwen9B+6.72+4.19+9.49+9.67+7.52
LLaVA-OV-1.54B+5.44+2.30+6.46+6.21+5.10
LLaVA-OV-1.58B+6.01+3.72+8.35+8.52+6.65
InternVL31B+1.38+1.41+1.43+1.25+1.37
InternVL39B+5.38+3.91+7.43+7.93+6.16
Molmo24B+5.17+2.65+6.28+6.63+5.18
Molmo29B+5.64+3.84+8.77+8.95+6.80

Table 6. VLM auditor transfer. All rows use the same replay protocol and differ only in the regional verifier.

Audit budget plot

Figure 4. Audit budget. More proposals improve repair quality; SEER stays above score-gated execution.

Cost and deployment scope

Residual failure cases showing mixed proposals, thin structures, and weak score support

Figure 5. Residual failure cases. Mixed proposals, thin structures, and weak score support can leave errors near boundaries. Additional examples appear in Appendix § 18.

Robustness checks. Joint threshold scaling changes the gain by at most 0.14 mIoU. One-factor sweeps give a largest per-dataset range of 0.083 mIoU. Replaying 8,000 decisions with three shuffled candidate orders gives a largest per-dataset decline of 0.078 mIoU. These checks use cached verifier outputs and leave the remaining pipeline fixed.

Failure boundary. Residual errors concentrate in mixed proposals, thin structures, candidate misses, and fragmented score support. The gate and conservative route reduce harmful writes, while some difficult regions remain unchanged.

Proposal and execution audit. We further audit 2,000 selected SAM2 proposals, with 500 from each mechanism benchmark. Ground truth is used only for this post-hoc analysis after proposal selection, VLM decisions, and execution have been fixed. Table 8 separates proposal quality from the behavior of the executable gate.

DiagnosticObservationExecution outcome
GT purity ≥0.95\ge0.951,732 / 2,000 (86.6%)+0.466 mean mIoU
GT purity < 0.7071 / 2,000 (3.55%)43 / 71 filtered
Low-purity bin effect−0.0055 / +0.0018Near-zero aggregate effect
Correct high-confidence decisions1,3971,061 base confirmations
Correct non-base directions336327 executed (97.32%)
Empty supported execution09 safely retain Y0Y^0

Table 8. Proposal quality and execution coverage. Purity and semantic correctness are analysis-only measurements over the fixed audit pool.

The useful gain is concentrated in high-purity proposals, while most low-purity proposals are suppressed and have negligible aggregate effect. Among correct non-base directions, the score-side executor applies 327 of 336 decisions; the remaining nine preserve the initial prediction, and none fails through an empty support. This analysis addresses proposal contamination and overly conservative execution separately.

Conclusion and limitations

SEER frames OVSS repair as executable region propagation after a fixed prediction has been produced. It couples closed-set VLM evidence with OVSS score-space execution: the verifier identifies actionable regional labels, and the segmenter decides their pixel support. This evidence-to-execution design supports high-recall propagation on coherent regions while using conservative support near uncertain boundaries. Across eight benchmarks, cross-base plug-in tests, and protocol-matched controls, the experiments show consistent repair gains with controlled proposal-level intervention cost. The controls attribute the gain to the coupling of closed-set regional evidence and score-controlled execution; proposals, crop classification, and smoothing each remain below the full system. The complete system is intended for offline or selective repair. It adds 9.3% cached or 19.3% latency and uses 18.7GB peak verifier memory in the measured configuration. Proposal misses, absent candidates, verifier errors, and weak score support remain important limitations, especially for thin or mixed regions. Future work will study lighter auditing, stronger proposal coverage, and more robust boundary execution.

Reproducibility Statement

Appendix C documents the execution contract, prompts, decoding settings, preprocessing, dataset mappings, and evaluation protocol. It also reports matched-input controls, threshold and candidate-order sensitivity, verifier and proposal-source transfer, and complete resource measurements. All benchmarks and frozen models are public, and the same parameter profile is used throughout.

SEER: Semantic Evidence for Executable Region Propagation in Open-Vocabulary Semantic Segmentation Supplementary Materials

Supplementary Overview

The main paper reports the compact experimental chain supporting the central claim that SEER converts closed-set regional evidence into executable OVSS label propagation. The supplementary material gives the extended diagnostics behind that chain and uses the same terminology as the main text. Coherent region propagation applies a certified label to the full proposal support. Boundary-conservative propagation restricts the support to score- and margin-supported pixels. Adaptive propagation chooses between these regimes based on evidence form and predicted risk.

Theory and Algorithmic Summary

This section gives the formal reading behind the main text. It does not claim a universal performance guarantee; instead, it shows why a VLM-assisted OVSS repair action naturally decomposes into a label-space direction and a pixel-space projector.

Notation. Let Ω\Omega be the pixel lattice, n=∣Ω∣n = |\Omega|, and C={1,…,C}\mathcal{C} = \{1, \ldots, C\} be the active OVSS label space. At repair step tt, the prediction map is Yt∈CnY^t \in\mathcal{C}^n and its one-hot matrix is Zt∈{0,1}n×CZ^t \in\{0,1\}^{n \times C}, where the pp-th row is zpt=eYptz_p^t = e_{Y_p^t}. For analysis only, let Y⋆Y^\star be the ground-truth map, gp=Yp⋆g_p = Y_p^\star, and Z⋆Z^\star its one-hot matrix. A local repair action is

r=(ℓ,W),ℓ∈C,W⊆Ω,(15)r = (\ell, W), \qquad\ell\in\mathcal{C}, \quad W \subseteq\Omega, \tag*{(15)}

where ℓ\ell is the target label and WW is the set of pixels to be changed. Let 1∈Rn\mathbf{1} \in\mathbb{R}^n be the all-one vector and let PW=diag⁡(1W)P_W = \operatorname{diag}(\mathbf{1}_W) be the diagonal projector onto pixels in WW. Applying rr is the linear update

Zt+1=Tℓ,W(Zt)=Zt+PW(1eℓ⊤−Zt).(16)Z^{t+1} = T_{\ell,W}(Z^t) = Z^t + P_W(\mathbf{1}e_\ell^\top- Z^t). \tag*{(16)}

Thus eℓe_\ell is a direction in label space, while PWP_W is a projector in pixel space. This separation is the algebraic object that SEER operationalizes.

Proposition B.1 (Exact benefit–risk decomposition). Define the pixel-error loss L(Z)=n−⟨Z,Z⋆⟩FL(Z) = n - \langle Z, Z^\star\rangle_F. For any action r=(ℓ,W)r = (\ell, W),

ΔL(r):=L(Tℓ,W(Zt))−L(Zt)=H(ℓ,W)−B(ℓ,W),(17)\Delta L(r) := L(T_{\ell,W}(Z^t)) - L(Z^t) = H(\ell, W) - B(\ell, W), \tag*{(17)}

where

H(ℓ,W)=∑p∈W1[Ypt=gp]1[ℓ≠gp],B(ℓ,W)=∑p∈W1[Ypt≠gp]1[ℓ=gp].(18)H(\ell, W) = \sum_{p \in W} \mathbf{1}[Y_p^t = g_p]\mathbf{1}[\ell\ne g_p], \qquad B(\ell, W) = \sum_{p \in W} \mathbf{1}[Y_p^t \ne g_p]\mathbf{1}[\ell= g_p]. \tag*{(18)}

Here H\mathcal{H} counts pixels harmed by the action and B\mathcal{B} counts pixels repaired by the action.

Proof. Starting from the definition of LL, the change in error is

ΔL(r)=(n−⟨Zt+1,Z⋆⟩F)−(n−⟨Zt,Z⋆⟩F)=−⟨Zt+1−Zt,Z⋆⟩F=−⟨PW(1eℓ⊤−Zt),Z⋆⟩F=⟨PW(Zt−1eℓ⊤),Z⋆⟩F.(19)\begin{aligned} \Delta L(r) &= \left(n - \langle Z^{t+1}, Z^\star\rangle_F\right) - \left(n - \langle Z^t, Z^\star\rangle_F\right) \\ &= -\langle Z^{t+1} - Z^t, Z^\star\rangle_F \\ &= -\langle P_W(\mathbf{1}e_\ell^\top- Z^t), Z^\star\rangle_F \\ &= \langle P_W(Z^t - \mathbf{1}e_\ell^\top), Z^\star\rangle_F. \tag*{(19)} \end{aligned}

Because PWP_W is diagonal, only rows in WW contribute:

ΔL(r)=∑p∈W⟨zpt−eℓ,egp⟩(20)\Delta L(r) = \sum_{p \in W} \langle z_p^t - e_\ell, e_{g_p} \rangle \tag*{(20)}
=∑p∈W(1[Ypt=gp]−1[ℓ=gp]).(21)= \sum_{p \in W} \left(\mathbf{1}[Y_p^t = g_p] - \mathbf{1}[\ell= g_p]\right). \tag*{(21)}

For a selected pixel pp, the summand is +1+1 exactly when Ypt=gpY^{t}_{p}=g_{p} and ℓ≠gp\ell\ne g_{p}, i.e., a correct pixel is overwritten by a wrong label; it is −1-1 exactly when Ypt≠gpY^{t}_{p}\ne g_{p} and ℓ=gp\ell=g_{p}, i.e., an incorrect pixel is repaired; otherwise it is 0. Grouping the positive and negative terms gives Eqs. (17)–(18).

Proposition B.1 explains why regional recognition alone cannot determine the value of a repair. Even if the label direction eℓe_{\ell} is semantically correct for part of a proposal, the action can still increase error when the projector PWP_{W} covers already-correct pixels. The central quantity is the net benefit produced by the pair (eℓ,PW)(e_{\ell},P_{W}).

Lemma B.2 (Regional evidence alone cannot identify the support projector). Let an evidence-only executor be any deterministic rule GG that receives only a proposal MM and a regional evidence tuple aa, and returns a support W=G(M,a)⊆MW=G(M,a)\subseteq M. Suppose aa certifies the same target label ℓ\ell for two local configurations but does not include dense OVSS scores, margins, or pixel ownership information. Then the support is not identifiable from (M,a)(M,a): there exist two configurations consistent with the same evidence for which no single WW minimizes ΔL\Delta L in both configurations.

Proof. Assume, for contradiction, that the support projector is identifiable from (M,a)(M,a), i.e., there exists a rule GG whose support is optimal for every configuration consistent with the same regional evidence. Choose a boundary pixel p∈Mp\in M whose regional evidence supports label ℓ\ell, and keep all pixels except pp identical across two configurations. Construct configuration AA so that pp is already correct and should not be edited:

Ypt,A=gpA≠ℓ.(22)Y^{t,A}_{p}=g^{A}_{p}\ne\ell. \tag*{(22)}

Construct configuration BB so that the same pixel is a residual error and should be changed to ℓ\ell:

Ypt,B≠gpB=ℓ.(23)Y^{t,B}_{p}\ne g^{B}_{p}=\ell. \tag*{(23)}

Both configurations share the same proposal MM and the same regional evidence tuple aa, because aa only states that the region provides evidence for label ℓ\ell; it does not encode whether the boundary pixel pp is already correct or belongs to the repairable part. Therefore GG must make the same inclusion decision for pp in both configurations.

If p∈G(M,a)p\in G(M,a), then by Eq. (18) this pixel contributes +1+1 to ΔL\Delta L in configuration AA, while excluding it would contribute 0 and strictly improve the action. Hence GG is not optimal in AA. If p∉G(M,a)p\notin G(M,a), then this pixel contributes 0 in configuration BB, while including it would contribute −1-1 and strictly improve the action. Hence GG is not optimal in BB. Both cases contradict uniform optimality. Thus regional evidence alone cannot identify the support projector.

Corollary B.3 (Evidence-to-execution factorization). A repair interface that aims to reduce ΔL\Delta L should use different information for the two factors of an action:

eℓ ←fvlm (I,Mi,Ai),PWi ←Γ (S,Y0,Mi,ℓ,ai).(24)e_{\ell}\ \xleftarrow{\mathrm{f_{vlm}}}\ (I,M_{i},A_{i}), \qquad P_{W_{i}}\ \xleftarrow{\Gamma}\ (S,Y^{0},M_{i},\ell,a_{i}). \tag*{(24)}

This is the formal motivation for SEER. The VLM verifies the semantic direction, while the OVSS score field and propagation policy determine the executable projector.

Closed candidates as a label-space projection. For proposal MiM_{i}, let bib_{i} be the dominant current label and let QiQ_{i} be the score-derived alternatives. The token uncertain represents abstention and is excluded from the class set. We define the executable candidate set

Ai+={bi}∪Qi⊆C.(25)\mathcal{A}^{+}_{i}=\{b_{i}\}\cup Q_{i}\subseteq\mathcal{C}. \tag*{(25)}

It induces a coordinate subspace and projection operator

Ui=span⁡{ec:c∈Ai+}⊆RC,ΠUiv=∑c∈Ai+⟨v,ec⟩ec.(26)\mathcal{U}_{i}=\operatorname{span}\{e_{c}:c\in\mathcal{A}^{+}_{i}\}\subseteq\mathbb{R}^{C}, \qquad\Pi_{\mathcal{U}_{i}}v=\sum_{c\in\mathcal{A}^{+}_{i}}\langle v,e_{c}\rangle e_{c}. \tag*{(26)}

If vi∈RCv_{i}\in\mathbb{R}^{C} denotes the verifier’s latent regional compatibility over the active vocabulary, a full-vocabulary verifier searches arg⁡max⁡c∈Cvi,c\arg\max_{c\in\mathcal{C}}v_{i,c}, while SEER searches only within the projected subspace:

ℓ^i∈arg⁡max⁡c∈Ai+⟨ΠUivi,ec⟩,or abstains with uncertain.(27)\hat{\ell}_{i}\in\arg\max_{c\in\mathcal{A}^{+}_{i}}\langle\Pi_{\mathcal{U}_{i}}v_{i},e_{c}\rangle,\quad\text{or abstains with uncertain.} \tag*{(27)}

This makes the closed candidate interface more than prompt shortening. It reduces the executable action directions from CC to ∣Ai+∣|\mathcal{A}_i^{+}|, and every non-abstained direction is both legal in the active label space and supported by the OVSS score tensor. The evidence-interface ablations in Table 3 and Table 12 empirically support this projection view.

InterfaceVOC21PC60CityADEAvg.
Initial prediction72.8844.0147.4128.6348.23
Free-form label75.7044.0147.4128.6348.94
Full label list78.8544.5653.4335.1953.01
Closed top-178.5645.3955.3637.0654.09
Closed top-279.8246.2354.2037.5654.45
Closed top-379.6048.2056.9038.3055.75

Table 12. Evidence interfaces. Absolute mIoU for the main gain table.

Propagation as constrained projector selection. For a certified label ℓ\ell, define the score-feasible support family

Pi(ℓ;S,Y0)={PW:W⊆Mi,ρ(p,ℓ,bi)=1 for all p∈W},(28)\mathcal{P}_i(\ell; S, Y^0) = \{P_W : W \subseteq M_i, \rho(p,\ell,b_i) = 1 \text{ for all } p \in W\}, \tag*{(28)}

where ρ\rho is the score-support predicate in Eq. (14). If Z⋆Z^\star were known, the best support within this family would solve

Pi⋆(ℓ)∈arg⁡min⁡PW∈Pi(ℓ;S,Y0)⟨PW(Zt−1ϵℓ⊤),Z⋆⟩F.(29)P_i^\star(\ell) \in\arg\min_{P_W \in\mathcal{P}_i(\ell;S,Y^0)} \left\langle P_W(Z^t - 1\epsilon_\ell^\top), Z^\star\right\rangle_F. \tag*{(29)}

Since Z⋆Z^\star is unavailable at inference time, SEER uses SS, margins, evidence form, and risk predicates as an executable surrogate for selecting PWP_W. Coherent propagation chooses a larger projector after strong certification, while score-grounded and boundary-conservative propagation choose smaller projectors when score support is ambiguous. The resulting action is therefore a constrained selector in the OVSS score space.

Algorithm 1 SEER inference. Regional evidence is certified before pixel-level propagation.

Require: image II, fixed prediction Y0Y^0, score tensor SS, label space C\mathcal{C}, SAM2 proposal bank M\mathcal{M}, audit budget KK

Ensure: repaired prediction YY

  1. Y←Y0Y \leftarrow Y^0

  2. R(I)←TopK⁡Mi∈Mπ(Y0,S,Mi)\mathcal{R}(I) \leftarrow\operatorname{TopK}_{M_i\in\mathcal{M}} \pi(Y^0,S,M_i)

  3. for each Mi∈R(I)M_i \in\mathcal{R}(I) do

  4. bi←mode⁡p∈MiYp\quad b_i \leftarrow\operatorname{mode}_{p\in M_i} Y_p

  5. Ai←{bi}∪Qi(S,Y,Mi)∪{uncertain}\quad\mathcal{A}_i \leftarrow\{b_i\} \cup Q_i(S,Y,M_i) \cup\{\text{uncertain}\}

  6. ai=(ℓ^i,conf⁡i,form⁡i)←fvlm(I,Mi,Ai)\quad a_i = (\hat{\ell}_i,\operatorname{conf}_i,\operatorname{form}_i) \leftarrow f_{\mathrm{vlm}}(I,M_i,\mathcal{A}_i)

  7. \quad if aia_i is not executable under Eq. (11) then

  8. \qquad continue

  9. \quad end if

  10. ℓi←ℓ^i\quad\ell_i \leftarrow\hat{\ell}_i

  11. Wi←Γ(S,Y,Mi,ℓi,ai)\quad W_i \leftarrow\Gamma(S,Y,M_i,\ell_i,a_i)

  12. Yp←ℓi\quad Y_p \leftarrow\ell_i for all p∈Wip \in W_i

  13. end for

  14. return YY

Implementation Details

Unified protocol. The protocol matrix in Table 9 records which factor changes in each study. The default state uses a fixed SAM2 proposal bank, closed top-3 candidates with uncertain, local Qwen3.5-9B deterministic decoding, the three-view prompt, and the default SEER propagation policy. The same coefficients and thresholds are used for all benchmarks without dataset-specific tuning. Closed top-kk denotes the current/base label plus up to kk non-base score-derived alternatives; uncertain is an abstention output and is not counted as a semantic candidate.

StudyScopeVaried factorHeld fixed
MainFull validationBackbone; benchmarkDefault verifier, prompt, candidates, policy
ComponentsFour benchmarksModule ablationPrediction, proposals, candidates, parser, split
InterfaceDefault audit poolResponse formatPrediction, proposals, executable gate, policy
ViewsDefault audit poolVisual contextCandidates, verifier, executable gate, policy
Cross-baseFull pipelineOVSS baseSame coefficients, thresholds, verifier, and rules
BudgetRanked prefixesAudit budgetCached evidence, executable gate, policy

Table 9. Unified protocol. Each row lists the experimental scope, the factor varied, and the quantities kept identical.

ComponentFixed rule/valueRole in execution
Priority scoreπi=1.20log⁡(1+100αi)+qi+0.75max⁡(δi,−0.35)+0.45λi+0.20ηi−0.90ri\pi_i = 1.20 \log(1 + 100\alpha_i) + q_i + 0.75 \max(\delta_i, -0.35) + 0.45\lambda_i + 0.20\eta_i - 0.90r_iRanks proposals before sparse VLM audit; weights are fixed once and not benchmark-fitted.
Risk predicateObject-like candidate/base, ri>0.20r_i > 0.20, or αi<0.015\alpha_i < 0.015Blocks broad writes for risky, small, or boundary-sensitive proposals.
Score support(τs,τm,τl)=(0.02,0.20,0.25)(\tau_s, \tau_m, \tau_l) = (0.02, 0.20, 0.25) in Eq. (14)Chooses score-compatible pixels for conservative propagation.
Propagation routeBroad support for coherent low-risk evidence; conservative support for object, boundary, small, or risky evidenceKeeps boundary evidence executable but restricts writes to selected support.
Parser and gateInvalid JSON, out-of-set label, uncertain, base label, low/medium confidence, mixed, or unclearAbstains with no pixel update; deterministic parser, no retry.
VLM auditLocal Qwen3.5-9B; three-view prompt; constrained JSON; deterministic decoding; 96-token capOne audited proposal gives one sparse VLM call; no closed oracle model is used.
View builderFull image width 640; local crop fits 4482448^2; tight crop fits 3362336^2Uses the same global, neighborhood, and tight views for every audit.
SAM2 proposalsHiera-L fp16; points 8; IoU/stability 0.4; multimask off; crop layers 0; NMS 0.7Supplies spatial domains only, never semantic predictions.
Audit budget2,000 unique audits per benchmarkKK counts VLM calls, not images or labels.

Table 10. Deterministic execution contract. Fixed constants and routing rules are shared by all benchmarks; invalid or abstained audits never modify pixels.

SEER uses prediction-side quantities free from inference time. The SAM2 proposal bank is precomputed and fixed in the main protocol to match the CorrCLIP-style evaluation setup; Table 11 varies only this spatial-domain source. Proposals define candidate supports for auditing and propagation, while semantic labels come from the OVSS score tensor and closed-set regional evidence. Fixing SAM2 keeps the protocol aligned and isolates the repair layer itself.

Mask generatorVOC21VOC20PC59PC60CityADEStuffObjectAvg.
Mask2Former Cheng et al. (2022)78.593.554.847.653.936.740.154.657.5
SAM2 Ravi et al. [34]79.694.255.248.256.938.338.652.958.0
EntitySeg Qi et al. (2023)81.194.857.849.758.140.639.654.259.5
SAM3 Carion et al. [7]82.395.658.650.557.940.040.456.560.2

Table 11. Proposal-source diagnostic. The main protocol uses SAM2 to align with the CorrCLIP-style configuration; this diagnostic varies only the mask generator that supplies spatial domains.

This diagnostic supports two points. First, Mask2Former, EntitySeg, and SAM3 all keep the pipeline functional when they provide reasonable spatial support. Second, stronger proposal coverage can lift the ceiling because proposals define the largest regions that can be audited and certified. The main experiments fix SAM2 for protocol alignment. SAM3 is strongest here, while its prompt-conditioned masks make it a diagnostic alternative outside the default protocol.

In the priority score, αi=∣Mi∣/∣Ω∣\alpha_i = |M_i|/|\Omega|, qiq_i is the dominant-label fraction, δi\delta_i is the candidate-minus-base score gap, λi\lambda_i is the low-margin fraction, ηi\eta_i is the mean entropy signal when available, and rir_i is the fixed risk-label fraction. For the view builder, d=max⁡(box width,box height)d = \max(\text{box width}, \text{box height}). The full view resizes the highlighted image to width 640, the local crop expands the proposal box by max⁡(32px,0.3d)\max(32\text{px}, 0.3d) before fitting it inside 448×448448 \times448, and the tight crop expands by max⁡(6px,0.3d)\max(6\text{px}, 0.3d) before fitting it inside 336×336336 \times336. The object-like routing set includes people/person, vehicles, animals, common furniture, and small handled objects when those names appear in the active dataset vocabulary. The structural risk-ratio set is fixed to common boundary-sensitive labels including background, wall, ceiling, floor, ground, road, building, tree, grass, sky, water, mountain, sidewalk, platform, rock, snow, wood, fence, door, window, curtain, light, sign, track, and shelves. These lists only affect routing and priority; they do not add candidate labels.

For a proposal MiM_i, the priority score combines four monotone signals: (i) the purity of the dominant baseline prediction within the proposal, (ii) proposal area normalized by image area, (iii) competition between the strongest non-baseline score and the baseline score, and (iv) the fraction of pixels predicted as risk-sensitive classes. This priority serves as a budget controller. Its coarse weights encode a fixed preference for impact, purity, score competition, uncertainty, and risk avoidance; they are not fitted per benchmark. It only orders proposals before VLM verification, and Appendix § 13 shows that ranking alone is insufficient without closed-set evidence and executable propagation.

For score-grounded propagation, Eq. (14) uses three fixed constants. τs\tau_s is a small tolerance that allows the target score to match the current baseline score, τm\tau_m defines low-margin policy, and τl\tau_l is the wider tolerance allowed only inside low-margin support. We set (τs,τm,τl)=(0.02,0.20,0.25)(\tau_s,\tau_m,\tau_l) = (0.02, 0.20, 0.25) once and keep the values unchanged across benchmarks. These rules form a deterministic execution contract: semantic evidence proposes a legal direction, while score support and risk predicates decide where the direction may write. Boundary-conservative propagation additionally requires low margin and is triggered when score support is weak after semantic certification. The default adaptive policy selects coherent propagation for certified low-risk regions and falls back to score-grounded or boundary-conservative propagation for object-like, boundary-like, large-risk, or high-risk proposals.

This support-threshold replay varies fixed constants around their default values. Using the main closed top-3 reference, gains are +6.72, +4.19, and +9.67 on VOC21, PC60, and ADE. Joint scaling shifts the replayed gains by at most 0.14 mIoU; PC60 and ADE move by 0.03 and 0.00. One-factor sweeps over τs\tau_s, τm\tau_m, and τl\tau_l change the four-dataset mean after-mIoU by 0.00007, 0.02093, and 0.01554, respectively; the largest range on any individual dataset is 0.08291. The audit interface, cached VLM evidence, parser, and propagation policy stay fixed in every replay.

VLM decoding. All VLM verifier outputs used by the main mechanism studies are produced with deterministic decoding (do_sample=False; greedy decoding with zero or implicit temperature). The replay diagnostics therefore reuse cached verifier outputs exactly as generated, which removes sampling noise from the protocol and keeps the audit sequence reproducible. The threshold stability diagnostic in Figure 6 checks the fixed support thresholds around their default setting with joint and one-factor perturbations.

Support-threshold stability plot

Figure 6. Support-threshold stability. Joint scaling around the default constants (0.02, 0.20, 0.25) changes gains by at most 0.14 mIoU.

Closed-Set Evidence Diagnostics

The main paper reports evidence-interface gains using Qwen3.5-9B as the default verifier. The supplementary view adds absolute mIoU in Table 12, execution counts in Table 13, and VLM filtering behavior in Table 15. These diagnostics should be read together because a stronger interface is useful only when it remains executable under the OVSS score field. Boundary-sensitive COCO-Object is analyzed separately in the routing and propagation diagnostics because its stuff-to-object label collapse makes broad regional evidence less directly executable, so the same label can be semantically plausible yet spatially risky.

InterfaceGainExec.Ben. pxHarm. px
Free-form label+0.704553.49M0.04M
Full label list+4.784,193160.53M9.06M
Closed top-1+5.863,700174.80M3.56M
Closed top-2+6.223,951183.11M4.07M
Closed top-3+7.525,217220.68M5.11M

Table 13. Execution statistics. Closed evidence executes useful regions.

BenchmarkCand. presentRaw high-conf. acc.Executed precisionFiltered/no exec.Invalid/OOS
VOC21100.0%62.4%81.9%38.9%0.2%
PC6099.5%67.7%92.7%35.2%0.4%
City100.0%48.5%74.3%36.2%0.1%
ADE100.0%53.1%80.2%34.5%0.1%
Average99.9%57.9%82.3%36.2%0.2%

Table 15. Execution filtering of VLM evidence. Candidate, filtered, and invalid/OOS rates use audited proposals as the denominator; raw accuracy and executed precision are measured on their corresponding pre- and post-filter subsets.

The candidate-size diagnostic in Table 14 compares the default Base + 3 interface with smaller and larger candidate lists. Target-in-set coverage is an analysis-only quantity computed with validation labels; it is unavailable to the verifier and propagation policy at inference time. All outcome rates use audited proposals as the denominator. Adding candidates changes both the verifier routing and abstention pattern, so correct outputs need not increase monotonically with kk. The best gain comes from the balance between target coverage, repair outputs, execution filtering, and conservative exits. Top-3 increases repair and correct outputs while allowing more abstain/filter exits than Top-2, which is intended under richer candidate context.

InterfaceTarget-in-set cov.Sem. cand.Repair outputCorrect outputBase/no editAbstain/filterAvg. gain
Base + 1 alt.56.3%2.0041.3%40.7%50.1%8.6%+5.86
Base + 2 alt.85.2%2.9747.6%39.4%46.3%6.1%+6.22
Base + 3 alt.99.9%3.3558.8%46.4%28.7%12.5%+7.52

Table 14. Candidate-set size and verifier outcomes. Rates are averaged over the four mechanism benchmarks with 2,000 audited proposals each. Base + kk alt. denotes the current/base label plus up to kk non-base score-derived alternatives; uncertain is an abstention output. Target-in-set coverage is diagnosis-only. Repair and correct-output rates are normalized by all audited proposals; executed precision is reported separately in Table 15.

These diagnostics support treating the VLM as a regional evidence source that still requires execution filtering. High-confidence answers alone have limited calibration. The parser, candidate-set validity checks, abstention path, and execution gate substantially improve the precision of decisions that are allowed to affect pixels. Table 15 measures audited proposals before and after filtering, while Table 32 summarizes routed proposals; the denominators are therefore different.

BenchmarkUncertain/lowCandidate missCorrect high-conf.Conservative route
VOC214.8%0.2%89.6%5.4%
PC603.7%0.5%87.2%8.6%
City3.4%0.3%89.5%6.8%
ADE4.6%0.8%84.9%9.7%
Object6.9%0.7%81.6%10.8%

Table 32. Regional-evidence routing taxonomy. Rates are computed over routed audited proposals. Conservative routing records boundary-aware execution; residual pixel harm is reported in the preceding table.

Propagation Policy Diagnostics

The propagation replay is extended to all eight benchmarks in Table 16 and to pixel-level exposure in Table 17. These endpoint policies show why propagation choice matters, not that one support type is universally superior. Coherent region propagation is strong when verified regions correspond to semantic units; boundary-conservative propagation is useful when object boundaries or proposal noise make broad propagation risky. The adaptive SEER policy is evaluated separately in Table 28, where routing among strict, lenient, and default support gives the strongest overall trade-off.

VOC20VOC21PC59PC60CityADEStuffObjectAvg.
Initial prediction87.6272.8848.6844.0147.4128.6333.3846.1951.10
Coherent mIoU91.9277.3853.1846.9153.6135.0336.3849.1955.45
Coherent gain+4.30+4.50+4.50+2.90+6.20+6.40+3.00+3.00+4.35
Boundary mIoU90.7276.2851.9846.3151.9133.8335.6848.5954.41
Boundary gain+3.10+3.40+3.30+2.30+4.50+5.20+2.30+2.40+3.31

Table 16. Full propagation-policy replay over eight benchmarks. Gains are measured over the fixed prediction. Beneficial/harmful pixels are diagnostic statistics computed after evaluation.

VOC20VOC21PC59PC60CityADEStuff
Coherent harmful px0.021.321.1413.851.410.761.18
Boundary harmful px0.001.280.8513.290.670.470.58
Coherent beneficial px2.328.2250.9799.1390.5448.00125.65
Boundary beneficial px1.246.4835.8871.0938.6334.1384.63

Table 17. Pixel exposure of propagation policies. Harmful pixels are shown in millions; full mIoU gains, including COCO-Object, are reported in Table 16.

PolicyVOCPCADESupport
Strict+3.73+2.47+7.060.66M
Lenient+4.56+3.91+8.230.82M
SEER+6.72+4.19+9.671.14M

Table 28. Policy sensitivity. SEER gives the largest gains with broader certified support.

Region Prioritization Diagnostics

The priority module controls the audit budget. Its coefficients are coarse, fixed ordering preferences for area, purity, score competition, uncertainty, and risk exposure; they are not tuned per benchmark and are not used after a region has been audited. The replay in Table 18 reorders the same audited pool and reuses cached VLM decisions, isolating proposal ordering without new VLM calls. Area is useful, while the full priority is stronger on average. This supports a modest claim: prediction-side ranking exposes a compact auditable set, and executable evidence propagation produces the main gain.

BenchmarkRandomArea-onlyUncertainty-onlyCompetition-onlyFull priority
VOC21+0.53+4.55+1.79+3.70+6.72
PC60+0.27+2.12+2.00+3.39+4.19
City+1.32+4.73+1.64+3.55+9.49
ADE+1.17+4.90+3.42+4.75+9.67
Average+0.82+4.08+2.21+3.85+7.52

Table 18. Audited-pool priority replay. Values are mIoU gains over the fixed prediction when the same audited decisions are replayed under different proposal orderings.

Candidate-order stability. We replayed 8,000 cached decisions with the original candidate order and three independent shuffles. The corresponding after-mIoU values are 49.4254, 49.4820, 49.4558, and 49.4701. The largest decline on any individual dataset is 0.0778 mIoU. Candidate position therefore has little influence under deterministic decoding and the closed executable gate.

Executable Gate Diagnostics

The relaxed-gate diagnostic in Table 19 removes the target/legal certification constraint and allows broader executable actions. Relaxing the gate can increase recall, while also expanding the number of writeback actions substantially and making conservative containment harder to preserve. The accompanying increase in routed support supports the gate’s role as an executable certification constraint.

BenchmarkInitial predictionAfter relaxed gateGainExecuted actions
VOC2172.8878.58+5.705,024
PC6044.0149.59+5.587,572
City47.4156.56+9.151,955
ADE28.6338.21+9.581,975
COCO-Object46.1950.55+4.361,608

Table 19. Relaxing executable certification. Removing the target/legal gate increases recall but permits many more writeback actions.

Control Interfaces and Non-VLM Diagnostics

Protocol-matched controls are compared in Table 20, with pixel statistics in Table 21. The score tensor contains useful repair signal, but score-only rules do not produce regional semantic evidence. A CLIP crop classifier tests ordinary region recognition, score-grounded support removes SEER’s adaptive executable propagation, and Dense-CRF-style smoothing tests generic local regularization. All controls remain below SEER under the same fixed-prediction protocol.

MethodVOC21PC60CityADEAvg.
OVSS score rule+2.10+3.80+2.53+2.13+2.64
CLIP crop classifier+3.88+2.38+2.70+5.80+3.69
VLM score support+2.09+0.87+1.74+2.11+1.70
Dense-CRF smoothing+0.01-0.01+0.02+0.01+0.01
SEER+6.72+4.19+9.49+9.67+7.52

Table 20. Protocol-matched controls. Same prediction state; only the repair signal or execution rule changes.

ControlExec.Ben.HarmGain
Cons. score107.01.66M0.05M+0.74
Margin score768.318.51M3.44M+2.64
Closed evidence1,304.355.17M1.28M+7.52

Table 21. Pixel statistics. Four-benchmark averages over executed repairs.

The cost and resource measurements in Table 22, Table 23, and Table 24 are local reproducibility diagnostics and do not imply real-time inference. The default verifier is a locally deployable open VLM. The 0.42s/call value is the effective latency after sharded execution, while the 2.09s/call resource profile is an unsharded local single-call measurement. Each VLM call audits one proposal using the three-view prompt, and the weighted average is computed over 14,053 validation images. SEER is training-free and does not update OVSS weights; its extra work scales with the number of audited proposals.

BenchmarkImagesCalls/imgVLM s/imgProp. s/imgTotal s/img
VOC211,4491.380.580.0430.62
PC605,1040.390.160.0430.21
City5004.001.680.0431.72
ADE2,0001.000.420.0430.46
COCO-Object5,0000.400.170.0430.21
Weighted avg.14,0530.710.300.0430.34

Table 22. Training-free cost. Selective proposal-level auditing with effective sharded verifier latency.

Scales/callPeakWeights
0.8B0.532.4 GB1.63 GiB
4B1.799.4 GB8.68 GiB
9B2.0918.7 GB17.98 GiB

Table 23. Qwen3.5 resources. Unsharded local single-call profile.

ComponentTime / image
Proposal generation0.508s
Frozen OVSS base forward5.042s
VLM verifier audit0.424s
Propagation / writeback0.043s
Cold end-to-end total6.018s
SEER repair overhead0.468s

Table 24. End-to-end timing. Validation-weighted cost with the default verifier.

Cross-Base Plug-in Diagnostics

The main paper applies the same complete SEER pipeline to multiple OVSS bases. The full-validation per-benchmark results appear in Table 26, with bootstrap intervals summarized in Table 25. Coefficients, thresholds, candidates, verifier, parser, gates, and propagation rules are unchanged; only the base prediction and score cache come from the chosen OVSS method. This controlled plug-in test complements the full eight-benchmark comparison in Table 1.

OVSS baseΔavg\Delta_{\mathrm{avg}}Bootstrap 95% CI
ProxyCLIP+4.53[+4.08, +4.92]
ResCLIP+5.08[+4.55, +5.50]
Trident+5.93[+5.34, +6.41]
CorrCLIP+5.48[+4.86, +5.97]

Table 25. Cross-base uncertainty. Bootstrap intervals for the plug-in gains in Table 26.

OVSS baseBenchmarkBase+SEERΔ\DeltaΔavg\Delta_{\mathrm{avg}}
ProxyCLIPVOC2161.364.9+3.6+4.53
ProxyCLIPPC6035.339.5+4.2+4.53
ProxyCLIPCity38.142.6+4.5+4.53
ProxyCLIPADE20.226.0+5.8+4.53
ResCLIPVOC2161.165.2+4.1+5.08
ResCLIPPC6033.538.3+4.8+5.08
ResCLIPCity35.941.1+5.2+5.08
ResCLIPADE18.024.2+6.2+5.08
TridentVOC2167.173.0+5.9+5.93
TridentPC6038.644.0+5.4+5.93
TridentCity42.948.8+5.9+5.93
TridentADE21.928.4+6.5+5.93
CorrCLIPVOC2174.879.3+4.5+5.48
CorrCLIPPC6044.247.7+3.5+5.48
CorrCLIPCity49.455.0+5.6+5.48
CorrCLIPADE26.935.2+8.3+5.48

Table 26. Full-validation cross-base plug-in test. The same SEER rules are used for each OVSS base; Δavg\Delta_{\mathrm{avg}} averages four benchmark gains.

Cross-base reading. This test is not intended to replace the full eight-benchmark comparison. It asks whether the same deterministic execution contract remains effective when the OVSS prediction and score cache change. All four bases improve on all four benchmarks, with average gains from +4.53 to +5.93 mIoU. This pattern indicates that the repaired residues are not idiosyncratic to CorrCLIP; the evidence-to-execution interface supports plug-in behavior when the new base provides a compatible score field.

Prompt, Policy, and Repair-Space Diagnostics

This section audits the repair mechanism from three angles. Prompt and policy sensitivity are reported in Table 27 and Table 28; proposal-space headroom is measured with validation-label oracles in Table 29; residual harm and routing behavior are summarized in the remaining diagnostics.

PromptVOC21PC60ADEAvg.
Concise+3.75+3.26+2.58+3.20
Neutral+5.98+2.73+3.49+4.07
Default+6.72+4.19+9.67+6.86

Table 27. Prompt schemas. The default prompt has the best average.

BenchmarkSEER closedOracle top-500Oracle full poolSEER/full
VOC21+6.72+13.01+16.380.41
PC60+4.19+2.33+6.560.64
City+9.49+13.92+21.740.44
ADE+9.67+6.37+19.540.49
Average+7.52+8.91+16.050.50

Table 29. Proposal-space repair headroom. Oracle rows are analysis-only diagnostics, not inference-time baselines. The full-pool oracle is the headroom row; the top-500 oracle is budget-limited and not a strict upper bound.

Prompt templates. All prompts use the same legal candidate set AiA_i and the same parser; only the amount of visual context and instruction detail changes.

Concise. “A region is highlighted in the image. Candidate labels are AiA_i. Choose exactly one candidate label or uncertain. Return JSON only: {label, confidence}.”

Neutral. “Inspect the highlighted region. The legal labels are AiA_i; do not use outside labels. Decide whether the region visually matches one candidate or should be uncertain. Return JSON only: {label, confidence, region_form}, where region_form is one of coherent, boundary, mixed, or unclear.”

Default. “You are verifying evidence for open-vocabulary semantic segmentation repair. You are shown the global image, a local neighborhood crop, and a tight crop of the same highlighted region. Use global context for scene semantics, local context for surrounding boundaries, and the tight crop for appearance. Candidate labels are AiA_i and include uncertain. Do not invent labels. Return JSON only: {label, confidence, region_form}. Use uncertain if the evidence is weak, mixed, outside the candidate set, or not executable as a coherent region.”

The oracle rows quantify remaining repair headroom inside the same proposal and action space. The full-pool oracle is the strict headroom diagnostic, while the top-500 oracle preserves the audit budget and can be lower than SEER when the budgeted oracle misses actions found by the default replay. Across the four mechanism benchmarks, SEER recovers about half of the full-pool oracle gain on average. COCO-Object is excluded because broad full-pool writes are not a safe upper bound in boundary-sensitive object settings.

BenchmarkImagesHarmed img.Med. lossP75 lossMean gainNeg. cls.
VOC211,4495.8%3.408.70+4.700
PC605,1040.9%3.268.95+2.830
City5005.0%0.431.92+6.180
ADE2,0003.8%0.822.47+2.915

Table 30. Image/class harm. Harmed-image rates and losses are measured after SEER repair; negative classes count class-IoU drops.

BenchmarkHarm/mod.Support spillSem. error
ADE2.1%69.8%30.2%
City2.1%72.4%27.6%
PC602.6%57.2%42.8%
VOC211.4%58.7%41.3%

Table 31. Residual harm sources. Harmful-pixel rates are measured among modified pixels; source shares are normalized only over residual harmful pixels.

The harm diagnostics address whether average mIoU gains hide systematic damage. Harmed-image rates stay low, no negative classes appear on VOC21, PC60, or City, and ADE has five small negative class deltas, led by stairway −0.57-0.57, ball −0.53-0.53, and base −0.32-0.32. Residual harmful pixels are sparse among modified pixels. Their sources split between support spill, where plausible evidence crosses ambiguous boundaries, and semantic error, where the regional label is wrong. This pattern supports the need for both closed evidence filtering and conservative score-space execution.

Limitations. SEER is a plug-in repair layer over a fixed OVSS prediction, so the complete-system comparison reports the accuracy-cost trade-off of adding frozen proposal-level regional auditing to an OVSS predictor, while matched-input diagnostics isolate repair. Its priority, gate, and routing rules form a deterministic execution contract, fixed across datasets, bases, and auditors to enforce legal labels, high-confidence evidence, and score-compatible support. SEER remains shaped by proposal coverage, candidate recall, regional verifier errors, and latent score support; improving thin structures, mixed proposals, crowded boundaries, and candidate-set misses in boundary-sensitive images is an important direction for future work.

Visual Examples and Figure Data

Failure and boundary-sensitive cases. Figure 7 shows residual cases where regional recognition is easier than executable support. Crowded street, food/table, and small-object scenes expose four residual risks: over-confident verifier outputs, candidate-set misses, mixed proposals, and weak score-field support. In these cases, a proposal may cover several semantic parts, or the correct label may be visible only in fragmented score support. SEER reduces their impact through abstention and conservative routing, which can leave part of a region unchanged when support is unreliable. Such cases also show the risk of direct VLM pixel authority. Larger galleries in Figure 8, Figure 9, Figure 10, Figure 11, Figure 12, Figure 13, Figure 14, and Figure 15 provide held-out examples.

Image, ground truth, CorrCLIP, and SEER visual examples

Figure 7. Boundary-sensitive and residual failure cases. SEER can recover proposal-supported regions absent from the initial top-1 prediction, but errors remain when proposals cross semantic boundaries, candidate labels are ambiguous, or the OVSS score field is weak. These cases motivate the execution gate and conservative propagation: plausible regional VLM evidence still needs score-field-constrained pixel support.

ADE20K visual examples

Figure 8. ADE20K visual examples. SEER repairs local semantic residues in indoor and scene-centric images while preserving the surrounding dense prediction structure.

Cityscapes visual examples

Figure 9. Cityscapes visual examples. SEER targets structured road-scene residues such as missing stuff regions, fragmented objects, and label-space confusions in a fixed OVSS prediction.

PC59 visual examples

Figure 10. PC59 visual examples. The repairs are localized to semantically supported regions and keep unrelated objects and stuff categories unchanged.

Visual examples comparing Image, GT, CorrCLIP, and SEER predictions

Figure 11. PC60 visual examples. The fixed prediction often contains plausible but incomplete regions; SEER uses closed-set evidence to propagate executable labels where the OVSS score field supports them.

VOC20 visual examples showing Image, GT, CorrCLIP, and SEER outputs

Figure 12. VOC20 visual examples. Object-centric cases show that SEER repairs missing or confused object regions through closed-set evidence and score-space execution.

VOC21 visual examples showing Image, GT, CorrCLIP, and SEER outputs

Figure 13. VOC21 visual examples. SEER improves residual object and background errors under the active label space, complementing the quantitative gains reported in the main paper.

COCO-Stuff visual examples

Figure 14. COCO-Stuff visual examples. Stuff-heavy scenes highlight the value of label-space-aware evidence: SEER corrects broad semantic regions while retaining pixel-level execution in the OVSS model.

COCO-Object visual examples showing Image, GT, CorrCLIP, and SEER results

Figure 15. COCO-Object visual examples. Object-boundary cases are more conservative, but SEER still repairs reliable regions when the closed candidate and execution gate agree.

References

  1. [1]Xiang An, Yin Xie, Kaicheng Yang, Wenkang Zhang, Xiuw ei Zhao, Zheng Cheng, Yirui Wang, Songcen Xu, Changrui Chen, Chunsheng Wu, Huajie Tan, Chunyuan Li, Jing Yang, Jie Yu, Xiaoyao Wang, Bin Qin, Yumeng Wang, Zizhen Yan, Ziyong Feng, Ziwei Liu, Bo Li, and Jiankang Deng. Llava-onevision-1.5: Fully open framework for democratized multimodal training, 2025. URL https://arxiv.org/abs/2509.23661.
  2. [2]Sule Bai, Yong Liu, Yifei Han, Haoji Zhang, Yansong Tang, Jie Zhou, and Jiwen Lu. Self-calibrated clip for training-free open-vocabulary segmentation. IEEE Transactions on Image Processing, 2025.DOI
  3. [3]Luca Barsellotti, Roberto Amoroso, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. Training-free open-vocabulary segmentation with offline diffusion-augmented prototype generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3689–3698, 2024.arxiv.org/abs/2404.06542
  4. [4]Walid Bousselham, Felix Petersen, Vittorio Ferrari, and Hilde Kuehne. Grounding everything: Emerging localization properties in vision-language transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3828–3837, 2024.arxiv.org/abs/2312.00878
  5. [5]Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez. Zero-shot semantic segmentation. Advances in Neural Information Processing Systems, 32, 2019.arxiv.org/abs/1906.00817
  6. [6]Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1209–1218, 2018.arxiv.org/abs/1612.03716
  7. [7]Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, et al. Sam 3: Segment anything with concepts. arXiv preprint arXiv:2511.16719, 2025.arxiv.org/abs/2511.16719
  8. [8]Junbum Cha, Jonghwan Mun, and Byungseok Roh. Learning to generate text-grounded mask for open-world semantic segmentation from only image-text pairs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11165–11174, 2023.
  9. [9]Junyang Chen, Xiangbo Lv, Zhiqiang Kou, Xingdong Sheng, Ning Xu, and Yiguo Qiao. Logoseg: Integrating local and global features for open-vocabulary semantic segmentation. Proceedings of the AAAI Conference on Artificial Intelligence, 40(4):2886–2894, 2026. doi: 10.1609/aaai.v40i4.37279. URL https://ojs.aaai.org/index.php/AAAI/article/view/37279.DOI
  10. [10]Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pp. 801–818, 2018.arxiv.org/abs/1802.02611
  11. [11]Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in neural information processing systems, 34:17864–17875, 2021.arxiv.org/abs/2107.06278
  12. [12]Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2818–2829, 2023.DOI
  13. [13]Seokju Cho, Heeseong Shin, Sunghwan Hong, Anurag Arnab, Paul Hongsuck Seo, and Seungryong Kim. Cat-seg: Cost aggregation for open-vocabulary semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4113–4123, 2024.arxiv.org/abs/2303.11797
  14. [14]Christopher Clark, Jieyu Zhang, Zixian Ma, Jae Sung Park, Mohammadreza Salehi, Rohun Tripathi, Sangho Lee, Zhongzheng Ren, Chris Dongjoo Kim, Yinuo Yang, et al. Molmo2: Open weights and data for vision-language models with video understanding and grounding. arXiv preprint arXiv:2601.10611, 2026.arxiv.org/abs/2601.10611
  15. [15]Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213–3223, 2016.arxiv.org/abs/1604.01685
  16. [16]Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 111(1):98–136, 2015.DOI
  17. [17]Sina Hajimiri, Ismail Ben Ayed, and Jose Dolz. Pay attention to your neighbours: Training-free open-vocabulary semantic segmentation. In Proceedings of the Winter Conference on Applications of Computer Vision, pp. 5061–5071, 2025.arxiv.org/abs/2404.08181
  18. [18]Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.arxiv.org/abs/2410.21276
  19. [19]Dahyun Kang and Minsu Cho. In defense of lazy visual grounding for open-vocabulary semantic segmentation. In European Conference on Computer Vision, pp. 143–164. Springer, 2024.arxiv.org/abs/2408.04961
  20. [20]Chanyoung Kim, Dayun Ju, Woojung Han, Ming-Hsuan Yang, and Seong Jae Hwang. Distilling spectral graph for object-context aware open-vocabulary semantic segmentation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 15033–15042, 2025.arxiv.org/abs/2411.17150
  21. [21]Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4015–4026, 2023.arxiv.org/abs/2304.02643
  22. [22]Philipp Krähenbühl and Vladlen Koltun. Efficient inference in fully connected crfs with gaussian edge potentials. Advances in neural information processing systems, 24, 2011.arxiv.org/abs/1210.5644
  23. [23]Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. Clearclip: Decomposing clip representations for dense vision-language inference. In European Conference on Computer Vision, pp. 143–160. Springer, 2024a.arxiv.org/abs/2407.12442
  24. [24]Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. Proxyclip: Proxy attention improves clip for open-vocabulary segmentation. In European Conference on Computer Vision, pp. 70–88. Springer, 2024b.arxiv.org/abs/2408.04883
  25. [25]Minhyeok Lee, Suhwan Cho, Jungho Lee, Sunghun Yang, Heeseung Choi, Ig-Jae Kim, and Sangyoun Lee. Effective sam combination for open-vocabulary semantic segmentation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 26081–26090, 2025.arxiv.org/abs/2411.14723
  26. [27]Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7061–7070, 2023.arxiv.org/abs/2210.04150
  27. [28]Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European conference on computer vision, pp. 38–55. Springer, 2024.arxiv.org/abs/2303.05499
  28. [29]Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 891–898, 2014.
  29. [31]Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URL https://qwen.ai/blog?id=qwen3.5.
  30. [32]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748–8763. PmLR, 2021.arxiv.org/abs/2103.00020
  31. [33]Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guided dense prediction with context-aware prompting. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18082–18091, 2022.arxiv.org/abs/2112.01518
  32. [34]Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024.arxiv.org/abs/2408.00714
  33. [35]Carsten Rother, Vladimir Kolmogorov, and Andrew Blake. "grabcut" interactive foreground extraction using iterated graph cuts. ACM transactions on graphics (TOG), 23(3):309–314, 2004.
  34. [36]Tong Shao, Zhuotao Tian, Hang Zhao, and Jingyong Su. Explore the potential of clip for training-free open vocabulary semantic segmentation. In European Conference on Computer Vision, pp. 139–156. Springer, 2024.arxiv.org/abs/2407.08268
  35. [37]Yuheng Shi, Minjing Dong, and Chang Xu. Harnessing vision foundation models for high-performance, training-free open vocabulary segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 23487–23497, 2025.
  36. [38]Lin Sun, Jiale Cao, Jin Xie, Xiaoheng Jiang, and Yanwei Pang. Cliper: Hierarchically improving spatial representation of clip for open-vocabulary semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 23199–23209, 2025.
  37. [39]Shuyang Sun, Runjia Li, Philip Torr, Xiue Gu, and Siyang Li. Clip as rnn: Segment countless visual concepts without training endeavor. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13171–13182, 2024.
  38. [40]Feng Wang, Jieru Mei, and Alan Yuille. Sclip: Rethinking self-attention for dense vision-language inference. In European conference on computer vision, pp. 315–332. Springer, 2024.
  39. [41]Jing Wang, Huimin Shi, Quan Zhou, Qibo Liu, Suofei Zhang, and Huimin Lu. Dcp-clip: A coarse-to-fine framework for open-vocabulary semantic segmentation with dual interaction. arXiv preprint arXiv:2603.13951, 2026.
  40. [42]Ji-Jia Wu, Andy Chia-Hao Chang, Chieh-Yu Chuang, Chun-Pei Chen, Yu-Lun Liu, Min-Hung Chen, Hou-Ning Hu, Yung-Yu Chuang, and Yen-Yu Lin. Image-text co-decomposition for text-supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26794–26803, 2024.arxiv.org/abs/2404.04231
  41. [43]Monika Wysoczańska, Oriane Siméoni, Michaël Ramamonjisoa, Andrei Bursuc, Tomasz Trzciński, and Patrick Pérez. Clip-denoiser: Teaching clip a few dino tricks for open-vocabulary semantic segmentation. In European Conference on Computer Vision, pp. 320–337. Springer, 2024.
  42. [44]Yongqin Xian, Subhbrata Choudhury, Yang He, Bernt Schiele, and Zeynep Akata. Semantic projection network for zero- and few-label semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8256–8265, 2019.
  43. [45]Enze Xie, W enhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in neural information processing systems, 34:12077–12090, 2021.arxiv.org/abs/2105.15203
  44. [46]Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18134–18144, 2022.arxiv.org/abs/2202.11094
  45. [47]Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open-vocabulary panoptic segmentation with text-to-image diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2955–2966, 2023a.arxiv.org/abs/2303.04803
  46. [48]Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. Side adapter network for open-vocabulary semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2945–2954, 2023b.arxiv.org/abs/2302.12242
  47. [49]Yuhang Yang, Jinhong Deng, Wen Li, and Lixin Duan. Resclip: Residual attention for training-free dense vision-language inference. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 29968–29978, 2025.DOI
  48. [50]Dengke Zhang, Fagui Liu, and Quan Tang. Corrclip: Reconstructing patch correlations in clip for open-vocabulary semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 24677–24687, 2025.arxiv.org/abs/2411.10086
  49. [51]Hang Zhao, Xavier Puig, Bolei Zhou, Sanja Fidler, and Antonio Torralba. Open vocabulary scene parsing. In Proceedings of the IEEE international conference on computer vision, pp. 2002–2010, 2017.arxiv.org/abs/1703.08769
  50. [52]Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region-based language-image pre-training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16793–16803, 2022.
  51. [53]Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International journal of computer vision, 127(3):302–321, 2019.arxiv.org/abs/1608.05442
  52. [54]Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In European conference on computer vision, pp. 696–712. Springer, 2022.arxiv.org/abs/2112.01071
  53. [55]Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479, 2025.arxiv.org/abs/2504.10479
  54. [56]Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, et al. Generalized decoding for pixel, image, and language. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15116–15127, 2023a.arxiv.org/abs/2212.11270
  55. [57]Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. Advances in neural information processing systems, 36:19769–19782, 2023b.arxiv.org/abs/2304.06718

Paper details

vlnc:W_2rvs3htj
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents