betaFeedback

WorksComputer Science

When Chekhov's Gun Fires Anyway: Over-Realization in Text-to-Image Models

Abstract

In text-to-image generation, a target that the prompt mentions often appears in the generated image in a way the prompt does not call for. We call this failure over-realization. Mental-spaces theory points to the situations where it can happen: under a negation, inside what a character believes or imagines, as the vehicle of a figurative comparison, or behind a viewpoint that hides the target. Prior work has studied only negation in depth, and the conventional alignment and preference metrics do not penalize the failure. We build OverReal to measure it: OverReal-Gen, 698 prompts covering the four situations, on which we measure each generator's over-realization rate; and OverReal-Det, 2,841 generated images human-labeled for whether and how the target is over-realized, on which we verify the automatic judge that produces this rate. Our evaluation of eight text-to-image systems reveals that all of them over-realize, and that conventional metrics such as CLIPScore reward the failure rather than penalize it. Prompt expansion only partly mitigates the failure. Inside the generator, the diffusion backbone attends to a mentioned target almost as much as to a requested one. The cue words that should hold the target back receive less attention than ordinary words.

Introduction

If in the first act you have hung a pistol on the wall, then in the following one it should be fired. —Anton Chekhov

Chekhov’s principle says that a planted gun is meant to fire, and to fire at the right moment. We borrow it in the reverse direction: generative models have learned to fire every gun a prompt mentions, whether or not the prompt meant it to appear in the scene. We study this failure in text-to-image generation, and Figure 1 shows two examples. In Figure 1a the prompt mentions a letter that says “I miss you”: to render the words legibly, the model turns the page toward the camera instead of its reader, and the scene contradicts itself. In Figure 1b the prompt calls the coach as fierce as a tiger: the model renders the tiger onto the team’s emblem and banner, which is not what the comparison meant. In both cases the model fired a gun it should have fired differently or left hanging. We call this failure over-realization: a mentioned target appears in the generated image in a way the prompt does not call for.

A man reading a letter behind a desk

(a) “Tom is reading a letter behind the desk. The letter is from his mom, which says: I miss you.”

A coach on a football field with a tiger on the team emblem and banner

(b) “The coach is as fierce as a tiger.”

Figure 1. Two over-realizations, both generated by Gemini, with the prompt under each image. In (a) the model turns the page to face the camera: the words cannot reach the reader, so the scene contradicts itself. In (b) the model puts a tiger on the team’s emblem and banner, which is not what the comparison meant, but the image still holds together.

Mental-spaces theory [11] suggests where over-realization can happen: some expressions in a prompt open a space alongside the scene to be drawn, and a target placed in that space is not part of the scene itself. Such a space can be opened by a negation, by what a character believes or imagines, by a figurative comparison, or by a viewpoint that hides the target. A model can then over-realize in two ways: the target disrupts the image, as in Figure 1a, or settles silently into the scene in a way the prompt did not call for, as in Figure 1b.

Of the four situations above, only negation has been studied in depth [1, 26, 6]; the figurative and the perspectival case have each been studied within a narrow scope, as idioms [31] and as occlusion [33]. Each is treated as a separate defect, and no prior work gives a systematic account of when a model may over-realize. Nor does current evaluation target the failure: it rests on two requirements, “a mentioned target should appear in the image” [16, 23] and “an unmentioned target should not appear” [22], and neither covers the case where “a mentioned target does not necessarily appear”, so an over-realized image is scored as correct.

We hypothesize that these situations share one failure mode: the model attends to the mentioned target almost as it would to a requested one, and renders it. To measure over-realization across the four situations and to examine this hypothesis, we build OverReal. OverReal-Gen is a prompt set covering the four situations, in which the target is prone to over-realization; a system’s over-realization rate on it is our measurement. OverReal-Det is a set of generated images human-labeled for whether and how the target is over-realized; it verifies the automatic judge that produces this rate.

All eight systems we evaluate over-realize, on 41% to 73% of the images of OverReal-Gen. The proprietary systems have largely solved the templated negation prompts, but every system over-realizes at least 32% of the images in the other three situations, and the form of the failure follows the situation. The conventional alignment and preference metrics do not penalize the failure and in fact reward it. Prompt expansion helps mostly because the rewriter drops the target; whenever the generator sees the target word, it tends to put the target in the image.

Our contributions are (1) a taxonomy of over-realization, motivated by mental-spaces theory, covering four situations and two forms of the failure; (2) OverReal, a prompt set for measuring over-realization and a human-labeled image set for verifying automatic judges; and (3) an evaluation of eight text-to-image systems, with an analysis of how conventional metrics, prompt expansion and the diffusion backbone relate to over-realization. We release OverReal, the images and the labels.

Related Work

Text-to-image generation and its evaluation Current text-to-image systems consist of a text encoder and a diffusion transformer trained by flow matching [10, 5, 35] (Qwen Team 2025), and the deployed proprietary systems [25, 12, 18] expand the user’s prompt before generation [4, 13]. Faithfulness to the prompt is the central criterion of text-to-image evaluation, and it has been understood as getting mentioned content into the image: T2I-CompBench [16], TIFA {(citep}(Hu et al., 2023)}, Davidsonian Scene Graph {(citep}(Cho et al., 2024)} and VQAScore {(citep}(Lin et al., 2024)} decompose the prompt into elements or questions and reward an image for each one it contains, and methods that improve faithfulness pursue the same goal on the generation side {(citep}(Chefer et al., 2023)}. Every mention is an obligation in these protocols, so an image that shows something the prompt mentions but does not call for scores as correct, or better. Hallucination benchmarks {(citep}(Qin et al., 2024; Lim et al., 2025; Kasaei & Rohban, 2025)} measure content the prompt never mentions. However, the opposite failure, content that is mentioned and should stay out, has not been well-studied.

Negation and semantic leakage Among the cases in which a mentioned element should stay out of the image, negation is the one that has been studied most. Vision-language models largely ignore it {(citep}(Alhamoud et al., 2025; Conwell et al., 2024)}, and negation-aware training data improves CLIP {(citep}(Park et al., 2025; Cai et al., 2025)}; we build our existence-canceling items on such prompts. Semantic leakage has been studied in generation within a narrower scope: a property of one requested entity appears on another {(citep}(Ventura et al., 2026)}. Reasoning benchmarks touch two further cases in passing, idioms rendered literally {(citep}(Sun et al., 2025)} and objects that should be occluded {(citep}(Wang et al., 2026)}. No prior work gives a systematic account and analysis of where and how over-realization can happen.

Preliminaries

A generative model over-realizes when a target word or concept that a prompt mentions appears in the generated image in a way the prompt does not call for. Other ways an image can fail to match its prompt, unrelated to the target, fall outside the definition of over-realization. We now describe where over-realization can happen, and how a model over-realizes in these situations; Appendix § 12.§ 1 gives an example of each combination in Table 2.

FamilyDisruptive over-realizationSilent over-realizationIntegrated realizationWithheld realization
1. Existence-cancelingprompt: a bath room with sinks and mirrors, no horse anywhere in the scene target: horse model: OmniGen2——prompt: boston Terrier sitting on a couch with a shoe, and not a single sandwich target: sandwich model: GPT-Image-1.5
2. Mental-stateprompt: Maya at the ridge, believing that there is a fierce tiger at the ridge and being super afraid. target: tiger model: SD3.5-Largeprompt: Sitting in math class, the thirsty girl stares at the whiteboard, wondering if she can buy a bottle of apple juice as soon as class ends. target: apple juice model: SD3.5-Largeprompt: Sitting on the sofa one warm morning, the kid thought that there were bears outside at night, although he had never seen them. target: bears model: GPT-Image-1.5prompt: A woman waits at a train station, imagining her deceased friend arriving on the next train. target: friend model: OmniGen2
3. Figurativeprompt: His vest is as green as a cucumber. target: cucumber model: SD3.5-Largeprompt: The unexplained expense was a red flag during an otherwise routine audit meeting. target: red flag model: Nano Bananaprompt: The runner is as fast as a cheetah as he races toward the finish line. target: cheetah model: Nano Bananaprompt: His wrinkles were as layered as tree rings. target: tree rings model: GPT-Image-1.5
4. Perspectivalprompt: A man holds a whiteboard against his chest with “Good luck!” written on the hidden side target: Good luck model: SD3.5-Largeprompt: A closed batman cookie jar contains a gingerbread men target: gingerbread men model: SD3.5-Largeprompt: Tom is reading a letter. The letter is from his mom, which says: I miss you. target: I miss you model: GPT-Image-1.5prompt: The student covered the grade on his exam sheet with a mug, pretending it wasn’t a bad score. target: score model: Ideogram v3

Table 2. Taxonomy examples.

Where over-realization can happen

To locate where over-realization occurs, we use mental-spaces theory from cognitive linguistics {(citep}(Fauconnier, 1994)}. In its terms, a prompt describes not one but several spaces: the scene to be generated (the base space), and alongside it the user’s own assumptions, beliefs, memories and comparisons (the non-base spaces). A text-to-image model should render only the base space, but it sometimes fails to do so. Over-realization is content from a non-base space leaking into the base space in a way the prompt does not call for. We group these situations into the following families:

Family 1: existence-canceling The target is mentioned under an operator that cancels its existence in the described scene. Typical explicit cues include no, without and never. It is well studied {(citep}(Alhamoud et al., 2025; Park et al., 2025; Cai et al., 2025)} and included for completeness; the target should not appear in the image.

Family 2: mental-state The target is mentioned inside the mental state of a person in the scene, so that it exists only in what that person believes, remembers or imagines. Typical explicit cues include believes, thinks, remembers and dreams. Unlike family 1, the target may legitimately be depicted, for instance inside a thought bubble or a hazy vignette, but it must not be shown as though it were physically present in the scene.

Family 3: figurative The target is mentioned as the vehicle of a figurative comparison: it characterizes something else in the scene rather than belonging to the scene itself. Typical cues include like, as … as, resembles and the suffix -like. The target should not appear in a photorealistic image, but where the image itself is stylized it may legitimately appear, for instance as a faded shape in the background of a poster.

Family 4: perspectival The target is mentioned as something that may not be visible from the viewpoint of the image: it lies behind a barrier, or on a surface turned away from the camera. Typical explicit cues include hidden, inside, closed and facing away. The target may be brought into view by a camera angle that does not contradict the scene, introduced through a quote box, or shown only in part or not at all.

How models over-realize

In these situations a model may over-realize the target or handle it correctly. We divide over-realization into two kinds according to whether the picture still holds together as a rendering of the requested scene:

Disruptive over-realization The target appears, and its appearance stops the picture from making sense: the scene contradicts itself or the prompt, or no longer reads as the requested scene from an ordinary view. In Figure 2(a) the coach is given the tiger’s head: the vehicle of the comparison has displaced the person it described.

Four possible outcomes for the prompt “The coach is as fierce as a tiger.”

Figure 2. Four possible outcomes for the prompt “The coach is as fierce as a tiger.” The target word tiger is mentioned only as the vehicle of a figurative comparison, so it is not strictly requested in the scene. (b) is the outcome the model actually produced. (a) and (b) are over-realization; (c) and (d) are correct.

Silent over-realization The target slips in while the picture holds together, typically in the background or in a detail. In Figure 2(b) the tiger appears on the team’s emblem and banner. Nothing in the picture is wrong on its own, but the emblem shows a tiger because the prompt mentioned one, not because the scene called for one. This is the warning level: the model has put the target in the scene in a way the prompt did not call for, and the scene happened to offer a harmless place for it. None of the evaluation methods we study detects this defect. Disruptive over-realization is an error; silent over-realization is a warning. We report the two shares separately throughout (Figure 3, Table 7), so a reader who does not count the silent cases can read the disruptive share alone.

Figure 3. Main results: over-realization rate of each generator by prompt family, on the raw prompts of OverReal-Gen (top) and on the in-the-wild prompts (bottom). Bar height is the over-realization rate; the dark part is disruptive (DO) and the light part silent (SO) over-realization; red bars are the proprietary generators, blue bars the open models. The full outcome shares, including integrated (I) and withheld (W) realization and the unprompted appearance rate uu, are in Table 7 (Appendix § 15).

GeneratorExistence-canceling DO↓SO↓I↑W↑u↓Mental-state DO↓SO↓I↑W↑u↓Figurative DO↓SO↓I↑W↑u↓Perspectival DO↓SO↓I↑W↑u↓
(a) OverReal-Gen
GPT-Image.02.00.00.98.02.35.26.28.11.08.22.29.16.33.05.54.15.03.28.05
Nano Banana.12.00.00.88.03.37.19.37.07.09.15.26.15.43.09.78.17.02.03.07
Ideogram.08.00.00.92.02.29.24.03.44.02.07.25.03.65.04.65.18.00.17.02
FLUX.1-dev.43.00.00.57.02.44.28.03.24.02.09.30.14.46.02.56.19.01.24.03
Qwen-Image.64.00.00.36.01.67.22.01.10.03.28.33.07.31.02.67.22.01.10.04
OmniGen2.51.00.00.49.01.64.22.03.11.02.19.27.08.46.03.64.16.01.18.03
SD3.5-Medium.34.00.00.66.00.66.21.01.12.02.09.31.06.54.02.67.11.00.22.02
SD3.5-Large.64.00.00.36.01.61.23.03.13.02.20.40.11.28.02.59.16.02.22.03
(b) In-the-wild prompts
GPT-Image.28.00.00.72—.21.12.64.03—.01.21.47.31—.59.23.00.18—
Nano Banana.38.00.00.62—.26.10.58.06—.01.24.35.40—.70.07.01.21—
Ideogram.55.00.00.45—.28.16.14.41—.02.27.26.45—.74.15.00.12—
FLUX.1-dev.70.00.00.30—.30.17.04.49—.02.29.14.54—.65.13.00.22—
Qwen-Image.72.00.00.28—.51.24.06.19—.04.26.36.34—.75.12.00.13—
OmniGen2.67.00.00.33—.42.18.04.36—.02.26.26.46—.66.09.00.25—
SD3.5-Medium.69.00.00.31—.33.20.03.44—.03.26.23.47—.63.10.00.26—
SD3.5-Large.70.00.00.30—.38.21.03.38—.02.26.29.43—.69.10.01.19—

Table 7. Main results: proportion of images under each outcome label (definition in § 3.2), namely disruptive over-realization (DO), silent over-realization (SO), integrated realization (I) and withheld realization (W), and the mean unprompted appearance rate (u, definition in § 5.3). (a) On the curated prompts of OverReal-Gen. (b) On the 397 in-the-wild prompts of § 6.1, one image per prompt and system; uu was not measured there.

A generation is correct when it does not over-realize the target and the picture matches the scene the prompt describes. We divide correct generations by whether the target appears.

Integrated realization The target appears, depicted coherently and consistently with the prompt. Figure 2(c) is a poster: the tiger looms behind the coach in a composition no one reads as a photograph, so it is taken as an attribute of the man. This is correct behavior wherever the target may legitimately be shown. However, in most of our cases the target must not appear in the scene as a physical entity (§ 3.1), which makes integrated realization hard to produce.

Withheld realization The target is left out and the picture is coherent without it. An image that is severely corrupted or contradicts the prompt is not withheld but other, and is excluded from the rates (Appendix § 17.§ 1). A mention does not oblige the model to depict, so this is as correct as an integrated depiction. In Figure 2(d) no tiger appears and nothing is lost: the coach’s face and stance carry the fierceness. Prior work treats a missing mention as a defect ([15] (Chefer et al. 2022)); here it is correct behavior wherever the target should not be shown.

Note that our definition covers only targets that appear because they were mentioned. A target can also belong to the scene, as a plane belongs to an airport, or come in by coincidence, as the tiger in Figure 2(b) would if the team were named after tigers. Such appearances are not over-realization; § 5.3 measures how often the target appears when it is not mentioned, to discount them.

OverReal Benchmark

OverReal-Gen: the generation benchmark

Data format Every entry of OverReal-Gen pairs a prompt with a target: the prompt describes a scene in which the target is prone to over-realization. Following § 3.1, we classify these situations into four main prompt families: existence-canceling, mental-state, figurative, and perspectival. Table 2 in Appendix § 12.§ 1 provides several examples.

Prompt collection Existence-canceling is well studied and its construction is direct, so we synthesize these prompts from templates following prior negation benchmarks [1, 9]. The mental-state, figurative, and perspectival prompts are all written by hand. Annotators write in natural language about everyday scenes, so that the prompts resemble those of a user; each annotator writes a prompt and names its target, Gemini generates three images, and the pair is kept if the annotator judges at least one image to over-realize the target and a second review confirms it. In total, 698 pairs enter OverReal-Gen (Table 3, Appendix § 13). OverReal-Gen is therefore a challenge set: the three hand-written families (not the templated existence-canceling one) are enriched for prompts on which a generator fails, so its rates measure how the systems handle prompts that are prone to over-realization. As a comparison free of this selection, § 6.1 also evaluates an in-the-wild set drawn from DiffusionDB [34] and Pick-a-Pic [20] by wording alone.

FamilyOverReal-GenOverReal-Det
RawExpandedGold sizeLabel distribution (%)Amb. sizeOverall agreement (κ\kappa)
SizeAvg. tokensSizeAvg. tokensDOSOIW
Existence-canceling20016.9400112.120432.40.50.067.2230.73
Mental-state7523.5150124.023953.618.011.317.21120.51
Figurative23716.4474116.493236.628.12.932.42990.65
Perspectival18623.1372118.787870.510.37.312.01540.72
Overall69819.11,396117.82,25351.217.65.226.05880.68

Table 3. OverReal statistics. Label distribution: share of the gold split labeled disruptive over-realization (DO), silent over-realization (SO), integrated realization (I) and withheld realization (W).

Prompt expansion Expanding the prompt with an LLM before generation is now a common part of the text-to-image pipeline. To control this stage, we collect prompts expanded from OverReal-Gen with the Qwen-Image rewriter (Qwen Team, 2025) and Ideogram Magic Prompt [17]. These 1,396 derived prompts form the expanded split (Table 3), which we use to analyze the effect of expansion on over-realization (§ 6.2).

OverReal-Det: the detection benchmark

Data format Every entry of OverReal-Det is a triple of a prompt, its target, and one generated image, and carries two independent annotations, each a set of labels from {disruptive, silent, integrated, withheld, other}, as defined in § 3.2.

Image collection The images come from two sources: (1) the pilot images produced during prompt collection (§ 4.1), and (2) images generated from the OverReal-Gen prompts with the models listed in § 5.

Annotation and splits Each image is labeled independently by two annotators following the guide in Appendix § 17.§ 2, who use two labels only for borderline images. An image whose two label sets intersect enters the gold split with the intersection as its label (a third annotator decides the three images left with two labels); an image whose sets are disjoint enters the ambiguous split, released with both label sets; images that both annotators mark other are discarded. This yields 2,253 gold and 588 ambiguous images, with an overall κ\kappa agreement of 0.68 (Table 3). Appendix § 13.§ 2 analyzes the ambiguous split.

Experimental Setup

We run eight generators, three proprietary and five open-weight (§ 5.1), on every prompt of OverReal-Gen (§ 4.1), and evaluate the generated images both with conventional metrics (§ 5.2) and from the perspective of over-realization (§ 5.3).

Text-to-Image Generators

We evaluate eight text-to-image generators. The proprietary generators are GPT-Image-1.5 [25], Ideogram v3 [18], and Nano Banana (gemini-2.5-flash-image) [12], while the open-weight models are the diffusion models FLUX.1-dev [5], Owen-Image (Owen Team, 2025), SD3.5-Medium and SD3.5-Large [10], and the unified understanding-and-generation model OmniGen2 [35]. An open-weight model receives the exact prompt; a proprietary generator rewrites it internally before generation, so the text that conditions its image is not under our control. Ideogram returns its rewritten prompt, which we record and reuse as a rewriter in § 6.2; GPT-Image and Nano Banana do not expose theirs. All images are generated at 1024 × 1024, the default resolution of most generators, and every open-weight model produces two images per prompt under fixed seeds.

Conventional evaluation

For each generator, family and prompt condition we score a fixed random sample of up to 100 images (7,050 in total) with five conventional text-to-image metrics. For text–image alignment we use CLIPScore [14] and the VQA-based VQAScore [23]; for learned human preference, PickScore [20] and HPSv2 [36]; and as an LLM judge, a VIEScore-style semantic-consistency rating [21] produced by GPT-5.2 [24] and Claude Opus 5 [2]; we report the Opus rating in the main text, since Opus shares no vendor with the evaluated generators, and the GPT-5.2 rating in Appendix § 15.§ 1 for reference. The scores of every generator are given in Table 6.

Existence-cancelingMental-stateFigurativePerspectival
CS↑VQ↑PS↑HP↑LJ↑CS↑VQ↑PS↑HP↑LJ↑CS↑VQ↑PS↑HP↑LJ↑CS↑VQ↑PS↑HP↑LJ↑
GPT-Image.67.8723.3.319.4.75.7822.1.338.2.69.8422.1.317.5.74.6622.4.295.6
Nano Banana.64.8222.9.288.7.71.7821.8.307.8.63.8321.6.287.5.71.7622.3.284.5
Ideogram.61.8222.7.298.3.66.6221.2.296.9.57.7321.1.276.9.70.7022.4.303.6
FLUX.1-dev.66.5922.9.305.8.73.6621.7.306.2.62.7021.4.295.5.73.6922.6.303.2
Qwen-Image.73.5123.2.315.2.75.7522.0.326.7.66.7921.7.306.3.77.7722.8.313.3
OmniGen2.70.5822.9.305.6.71.6421.5.325.8.63.7121.2.285.3.75.7222.3.303.0
SD3.5-Medium.69.6922.5.286.5.72.6521.4.296.0.62.6921.1.265.5.76.6921.9.283.2
SD3.5-Large.71.5022.9.305.0.73.6721.8.316.3.66.7121.6.295.4.75.7122.5.303.6

Table 6. Conventional evaluation of the eight systems on the raw prompts of OverReal-Gen: mean CLIPScore (CS), VQAScore (VQ), PickScore (PS), HPSv2 (HP) and LLM judge (LJ) per family.

Over-realization evaluation

Automatic labeling Evaluating 698 prompts on eight generators, under up to three prompt conditions and two seeds, yields tens of thousands of images, so exhaustive manual annotation is infeasible. We therefore label the images with a third-party model, Claude Opus 5 [2]. The model substantially agrees with human annotators (κ=0.70\kappa= 0.70) on whether a generator over-realizes or not over the test split of OverReal-Det. Details are given in Appendix § 14. All rates below are over the images that received one of the four labels; the 15 images (0.5%) labeled other are excluded.

Unprompted appearance rate A target can appear in a scene for reasons unrelated to its mention. To discount such coincidences from the over-realization rates, we also report the unprompted appearance rate uu of the target: we replace the target in the prompt with a blank (“___”), generate images from the edited prompt with the same settings, and count how often the target still appears. Ideally uu is close to zero.

Results and Analysis

Main results

Figure 3 shows the over-realization rate of three proprietary and five open generators (§5.1) on the raw prompts of OverReal-Gen and on a set of in-the-wild prompts; the full outcome shares are in Table 7 (Appendix § 15). Conclusions are given as follows.

All generators over-realize, and the rate depends on the prompt family. We first read the over-realization rate, the sum of the DO and SO shares; its prompt-level bootstrap 95% interval is within ±0.13 in every cell (Appendix § 15.§ 2). Over all prompts, weighting the four families by their prompt counts, each generator over-realizes 41% to 73% of the images. The proprietary generators rarely over-realize the templated existence-canceling prompts (at most 12% of the images), while the open models still do so on 34% to 64%. Every model fails on mental-state and perspectival prompts, over-realizing about half or more of the images; on perspectival prompts the proprietary generators are no better than the open ones, and Nano Banana over-realizes 95% of the images. Figurative prompts sit in between, at 32% to 62%. Counting only disruptive over-realization, every generator still over-realizes at least 29% of the mental-state and 54% of the perspectival images, while the figurative rates fall to 7% to 28% (Table 7).

The form of the failure depends on the family. The two forms of over-realization are not distributed alike. On perspectival prompts the failures are mostly disruptive: the hidden object or content is exposed, contradictory with the prompt or common sense. On figurative prompts, by contrast, silent over-realization accounts for 25% to 40% of the images and exceeds the disruptive form for all eight generators: the vehicle of the comparison is drawn into the scene as a harmless detail.

Unprompted appearance does not explain the rates. The unprompted appearance rate uu (Table 7) stays at or below 9% in every cell, and below 5% for the open models, far below the over-realization rates. The rates of Figure 3 are therefore not explained by the target belonging to the scene, and coincidental appearances inflate them only marginally.

Silent over-realization is a precursor of disruptive over-realization, except in the figurative family. Silent over-realization does not disrupt the image, but it is a warning sign of disruptive over-realization rather than a lucky form of correct realization: an image labeled silent is over-realized again on the second seed 90% of the time (disruptive 33%, silent 57%) and correct only 10% of the time (Appendix § 15.§ 3). The pattern holds for every open model and is strongest on perspectival and mental-state prompts; on figurative prompts a silent image is followed by another silent one 76% of the time and rarely turns disruptive, since the scene can usually hold the vehicle of the comparison.

Non-LLM conventional metrics reward over-realization and penalize withholding. Table 1 reports the mean score of each metric over the images with each label. A metric sensitive to over-realization would score the two correct outcomes (I, W) above the two over-realizations (DO, SO). The alignment (CS, VQ) and preference (PS, HP) metrics credit integrated images but give withheld images the lowest score in most cases: they reward the presence of the target, not its correct handling. Only the LLM judge scores withheld images above over-realized ones, in three of the four families. Within matched pairs of images from the same prompt and generator whose two seeds differ in outcome, CLIPScore, PickScore and HPSv2 score the over-realized image higher in 64% to 66% of the pairs (Appendix § 15.§ 4): the conventional metrics reward over-realization even with prompt and generator held fixed.

Existence-cancelingMental-stateFigurativePerspectival
CS↑CS\uparrowVQ↑VQ\uparrowPS↑PS\uparrowHP↑HP\uparrowLJ↑LJ\uparrowCS↑CS\uparrowVQ↑VQ\uparrowPS↑PS\uparrowHP↑HP\uparrowLJ↑LJ\uparrowCS↑CS\uparrowVQ↑VQ\uparrowPS↑PS\uparrowHP↑HP\uparrowLJ↑LJ\uparrowCS↑CS\uparrowVQ↑VQ\uparrowPS↑PS\uparrowHP↑HP\uparrowLJ↑LJ\uparrow
DO.72.3222.9.303.0.73.7121.8.326.4.69.7422.0.304.8.74.7422.5.303.6
SO—————.73.6921.8.306.7.63.7421.5.296.0.74.7222.3.293.5
I—————.74.8121.7.318.3.69.8621.9.296.9.76.8522.2.304.3
W.66.8522.9.308.8.67.5921.3.296.6.60.7521.2.276.8.67.5921.9.284.5

Table 1. Mean score of the conventional metrics over the images with each label, pooled over generators; cells with fewer than 10 images are left blank.

The conclusions hold on in-the-wild prompts. For comparison with the curated OverReal-Gen (§ 4.1), we take 397 prompts of the four families from DiffusionDB ([34]) and Pick-a-Pic ([20]), real user prompts selected by their wording alone (Appendix § 15.§ 6). The bottom row of Figure 3 shows that every generator still over-realizes in every family: pooled over generators, the rate is 60%, 51%, 28% and 80% on existence-canceling, mental-state, figurative and perspectival prompts, with the same family patterns as on the curated prompts, which run 0.1 to 0.3 higher on mental-state and figurative prompts and lower on existence-canceling ones. A human check of the automatic labels on 202 of the wild images gives κ=0.70\kappa= 0.70 over the four labels, so these rates rest on labels as reliable as those on OverReal-Det (Appendix § 15.§ 6).

The effect of prompt expansion

The proprietary generators expand the prompt to include more details before generating, so their advantage could come from either the prompt expansion or the generator. This section asks whether prompt expansion reduces over-realization. We compare five open models, with their own expansion disabled, on the raw prompts and on the prompts rewritten by the Qwen-Image official rewriter and by Ideogram Magic Prompt (Table 13, Appendix § 15.§ 5).

Generator + rewriterExistence-canceling DO↓SO↓I↑W↑Mental-state DO↓SO↓I↑W↑Figurative DO↓SO↓I↑W↑Perspectival DO↓SO↓I↑W↑
FLUX.1-dev (raw).43.00.00.57.44.28.03.24.09.30.14.46.56.19.01.24
+ Qwen rewriter.33 (−.10).00 (0).00 (0).67 (+.10).41 (−.03).25 (−.03).05 (+.02).29 (+.05).05 (−.04).21 (−.09).09 (−.05).65 (+.19).65 (+.09).20 (+.01).01 (0).14 (−.10)
+ Ideogram Magic.11 (−.32).00 (0).00 (0).89 (+.32).15 (−.29).32 (+.04).04 (+.01).49 (+.25).00 (−.09).20 (−.10).04 (−.10).76 (+.30).62 (+.06).15 (−.04).01 (0).22 (−.02)
Qwen-Image (raw).64.00.00.36.66.22.01.10.28.33.07.31.67.22.01.10
+ Qwen rewriter.81 (+.17).00 (0).00 (0).19 (−.17).50 (−.16).35 (+.13).10 (+.09).05 (−.05).14 (−.14).28 (−.05).12 (+.05).46 (+.15).72 (+.05).20 (−.02).00 (−.01).08 (−.02)
+ Ideogram Magic.08 (−.56).00 (0).00 (0).92 (+.56).26 (−.40).22 (0).06 (+.05).46 (+.36).04 (−.24).22 (−.11).07 (0).67 (+.36).64 (−.03).19 (−.03).01 (0).16 (+.06)
OmniGen2 (raw).51.00.00.49.64.22.03.11.18.26.08.45.63.16.01.18
+ Qwen rewriter.20 (−.31).00 (0).00 (0).80 (+.31).40 (−.24).31 (+.09).08 (+.05).21 (+.10).09 (−.09).18 (−.08).06 (−.02).67 (+.22).66 (+.03).18 (+.02).00 (−.01).16 (−.02)
+ Ideogram Magic.09 (−.42).00 (0).00 (0).91 (+.42).16 (−.48).27 (+.05).07 (+.04).50 (+.39).02 (−.16).14 (−.12).07 (−.01).76 (+.31).63 (0).14 (−.02).00 (−.01).23 (+.05)
SD3.5-Medium (raw).34.00.00.66.66.21.01.12.09.31.06.53.66.11.00.22
+ Qwen rewriter.27 (−.07).00 (0).00 (0).73 (+.07).47 (−.19).22 (+.01).05 (+.04).26 (+.14).09 (0).23 (−.08).12 (+.06).55 (+.02).68 (+.02).18 (+.07).01 (+.01).11 (−.11)
+ Ideogram Magic.07 (−.27).00 (0).00 (0).93 (+.27).23 (−.43).28 (+.07).00 (−.01).48 (+.36).02 (−.07).19 (−.12).04 (−.02).75 (+.22).62 (−.04).11 (0).00 (−.01).27 (+.05)
SD3.5-Large (raw).64.00.00.36.61.23.03.13.20.40.11.28.58.16.02.22
+ Qwen rewriter.34 (−.30).00 (0).00 (0).66 (+.30).43 (−.18).30 (+.07).04 (+.01).23 (+.10).13 (−.07).30 (−.10).09 (−.02).54 (+.26).67 (+.09).22 (+.06).00 (−.02).09 (−.13)
+ Ideogram Magic.11 (−.53).00 (0).00 (0).89 (+.53).27 (−.34).24 (+.01).02 (−.01).47 (+.34).03 (−.17).17 (−.23).03 (−.08).77 (+.49).69 (+.11).15 (−.01).01 (−.01).15 (−.07)

Table 13. Performance of prompt expansion.

Expansion lowers the rate but does not remove the failure. On existence-canceling, mental-state and figurative prompts, a rewrite lowers the over-realization rate in 29 of the 30 generator–rewriter pairs; the largest gains are on existence-canceling prompts, where the Ideogram rewrite brings the rate to 11% or below. The Ideogram rewrite is the more reliable of the two: it improves every cell of the three families and roughly halves the rates in the mental-state and figurative families, eliminating 58% of the over-realizations against 21% for the Qwen rewrite, which on existence-canceling prompts even raises the rate of its own generator, Qwen-Image, from 64% to 81%. Perspectival prompts are the exception: the Qwen rewrite raises the rate for all five generators, to 84% to 91%, and the Ideogram rewrite leaves it within a few points of the raw prompt. And even with the Ideogram rewrite, the open generators still over-realize about 48% of the mental-state prompts, 21% of the figurative ones and 79% of the perspectival ones.

The gain comes from dropping the target. We classified every rewritten prompt by its treatment of the target: dropped (no longer mentioned), explicitly excluded (stated to be absent), mentioned (named, but nothing calls for it to be shown), or explicitly depicted (stated to be shown); Figure 4 summarizes the distribution and Appendix § 15.§ 7 gives it in full. When a target is not meant to appear, Ideogram drops it whereas Qwen explicitly excludes it, and Figure 4(b) shows that only dropping substantially suppresses over-realization; Ideogram also recognizes more often that the target should not appear, dropping it in 41% of the mental-state and 55% of the figurative prompts, against 0% and 15% for Qwen. Neither rewriter reasons correctly about perspectival prompts: both keep the target in nearly every rewrite and describe it as visible in about half. The three treatments other than dropping still yield over-realization in 38% to 84% of the images, so the generator over-realizes on its own, and § 6.3 turns to its side.

Figure 4. What the rewriters do with the target. (a) Proportion of rewritten prompts in which the target is dropped, explicitly excluded, mentioned, or explicitly to be depicted. (b) Over-realization rate of the images generated from those prompts, by the four categories, pooled over rewriters and open generators. Full numbers in Table 16 and Table 17.

Attention to the target and the cue words

An open text-to-image model has two stages, a text encoder and a diffusion backbone that reads the encoder’s output while denoising the image. Here we read the attention inside the backbones of FLUX.1-dev, SD3.5-Large and Qwen-Image under the setting of § 5; the encoder is examined in Appendix § 15.8. To compare a target the prompt does not call for with a requested one, we write a plain-mention control for each prompt by hand, replacing the cue words of each family (no, imagines, like, hidden) by words that put the target into the scene, and generate it under the same seed: “His vest is as green as a cucumber” becomes “His vest is green, and he holds a cucumber” (more in Table 20). Table 21 in Appendix § 15.9 reports the share of attention on the target words, the cue words and the other words, per family and averaged. Three observations follow.

FamilyOriginal promptPlain-mention control
Existence-cancelinga cat sitting on a couch with a dog on the floor, with no book in sighta cat sitting on a couch with a dog on the floor, with a book in sight
Mental-stateTim sits alone at a classroom desk, remembering the toy robot he lost years ago.Tim sits alone at a classroom desk, holding the toy robot he got years ago.
FigurativeThe coach is as fierce as a tiger.The coach is standing next to a tiger.
PerspectivalA desk drawer is completely closed. Inside is a green frog toy.A desk drawer is pulled open. Inside is a green frog toy.

Table 20. Plain-mention controls: one example per family. The target word is kept; the words that keep it out of the scene are replaced by words that put it in.

ModelFamilyOriginal prompt target ata_tOriginal prompt cue aca_cOriginal prompt other aoa_oPlain-mention control target at′a'_tPlain-mention control replacing ar′a'_rPlain-mention control other ao′a'_oRatios ρt=at/at′\rho_t = a_t/a'_tRatios ρc=ac/ar′\rho_c = a_c/a'_rRatios γ=ac/ao\gamma= a_c/a_o
FLUX.1-devExistence-canceling4.04.05.65.13.85.4.791.15.71
Mental-state3.83.04.24.83.54.2.74.87.73
Figurative5.64.76.16.55.65.7.83.80.77
Perspectival3.63.74.43.73.64.5.981.04.83
All families4.23.85.15.04.14.9.83.97.76
SD3.5-LargeExistence-canceling2.41.43.12.81.83.0.85.82.46
Mental-state2.41.32.32.61.62.3.85.79.54
Figurative4.22.13.34.42.63.1.92.80.63
Perspectival2.11.92.42.12.02.4.99.97.81
All families2.81.72.83.02.02.7.90.84.60
Qwen-ImageExistence-canceling5.24.66.36.36.26.3.80.84.73
Mental-state3.72.75.14.03.35.0.90.83.54
Figurative5.14.27.85.44.87.7.92.86.54
Perspectival3.63.45.23.63.55.2.971.03.66
All families4.43.86.14.84.46.0.90.89.62

Table 21. Attention in the backbone, per family and averaged over the four families. aa: percentage of the image-to-text attention per token of the target words (t), the cue words (c) and the other words (o) under the original prompt; a′a': the same under the plain-mention control, where the cue words are replaced (r). The ratios pair the original prompt and its control under the same seed.

A mentioned target receives almost as much attention as a requested one. Under the original prompt, one token of the target words receives 2.8% to 4.4% of the image-to-text attention (ata_t), while under the control the same words receive 3.0% to 5.0% (at′a'_t). The ratio ρt\rho_t, 0.83 to 0.90, means that the target receives almost the same attention under the original prompt as under the control, although the control requests the target and the original only mentions it. Therefore, the backbone’s attention barely distinguishes a mentioned target from a requested one. Figure 10 (top row, first two columns) shows the map for the cucumber of Table 2: under both prompts the attention sits where the cucumber is drawn, on the man’s head under the original prompt and in his hand under the control.

Two items of Table 2 under four prompts, showing generated images and attention maps

Figure 10. Two items of Table 2 under four prompts, SD3.5-Large: the original prompt, its plain-mention control, the Qwen rewrite and the Ideogram Magic Prompt rewrite (top row: the image; bottom row: attention from the image tokens to the target words in the first denoising steps). Top: the cucumber. The Qwen rewrite keeps cucumber only as a color (“the deep, fresh hue of a ripe cucumber”); the backbone draws no cucumber and the map stays flat. The Ideogram rewrite drops the word, so there is no map. Bottom: the whiteboard whose “Good luck!” faces away from the camera. Both rewriters keep the words, both images show them, and under all four prompts the attention sits on the writing.

The cue words receive less attention than ordinary words. The cue words (no, imagines, like, hidden) receive 1.7% to 3.8% per token (aca_c), less than an ordinary word (γ=0.60\gamma= 0.60 to 0.76), and also less than the replacing words (with, holding, next to, open; ρc=0.84\rho_c = 0.84 to 0.97). The backbone’s attention, then, does not reflect what the cue words say: that the target need not appear in the image, and if it does, only in a form the prompt allows, such as a thought bubble, a stylized shape, a reflection or a viewpoint that brings it into sight.

Over-realization rate increases with more attention on the target words and less attention on the cue words. Figure 5 relates the attention share of target words and cue words to the over-realization. From the lowest to the highest interval of the figure, the over-realization rate rises with the share of the target words (0.44 → 0.77, 0.59 → 0.92 and 0.68 → 0.88 for FLUX.1-dev, SD3.5-Large and Qwen-Image) and falls with the share of the cue words (0.60 → 0.25, 0.72 → 0.59 and 0.69 → 0.53), the fall being steepest where the cue words receive an unusually large share. The over-realization rate thus tracks both attention shares, rising with the share of the target words and falling with the share of the cue words on all three backbones.

Figure 5. Histograms: the distribution over the attention of the target words (green) and of the cue words (red). Lines: the over-realization rate (OR rate) within each interval of the share; intervals with fewer than 12 images are not drawn.

Conclusion

We introduced over-realization, the appearance of a mentioned target in a generated image in a way the prompt does not call for, and drew on mental-spaces theory to organize the four situations where it can happen. OverReal measures it with a prompt set prone to over-realization and a human-labeled image set. All eight text-to-image systems over-realize outside negation, on the benchmark and on real user prompts; the form of the failure follows the family, and conventional metrics reward over-realization. Prompt expansion helps mainly when the rewriter drops the target; when the target word reaches the generator, its backbone attends to it almost as to a requested one and to the cue words less than to ordinary words. Faithful generation therefore requires a model that can decide on its own which mentioned target to leave out (Appendices § 12.§ 3 and § 15.11), and evaluation that credits it for withholding.

Limitations

Modality. Over-realization is not specific to text-to-image generation: a language model can realize a mentioned target in text, and a video generator in its own medium. This paper covers images, where the failure produces a visible contradiction and generation and evaluation are affordable; extending the taxonomy and the benchmark to other modalities is left for future work.

Subjectivity of the outcome labels. The four outcome labels are defined by the effect that the target’s appearance has on the image, and this effect is a matter of degree, in particular between disruptive, silent and integrated. The annotators did not always agree; Appendix § 12.§ 2 gives borderline examples and Appendix § 13.§ 2 counts the disagreements.

Ethics Statement

The prompts are written by the authors or synthesized from templates, and the images are generated by the evaluated models; no personal data is involved. The work studies a faithfulness failure of text-to-image generation and provides no means of attack.

Reproducibility Statement

OverReal-Gen and OverReal-Det are released with the annotations, the plain-mention controls and the expanded prompts. All generator settings are provided in § 5.1. The judges and the automatic annotator are identified by model version in § 5.2 and § 5.3, and the exact question text of the annotation protocol and the human annotation guide are given in Appendix § 17. All code for generation, annotation, probing and attention recording is released with the data.

AI Use Statement

Large language models are part of the evaluated pipeline and of the evaluation itself: Claude Opus 5 and GPT-5.2 serve as the LLM judge (§ 5.2) and Claude Opus 5 as the automatic annotator (§ 5.3); the Qwen-Image rewriter and Ideogram Magic Prompt produce the expanded prompts (§ 6.2). The authors used Claude Code as a coding and writing assistant; all analyses, claims and text were checked by the authors, who take full responsibility for the content.

Example of Generated Images

Taxonomy

Table 2 shows more examples of over-realization with different labels in each category. All images are produced by the generators of § 5.1 with prompts from OverReal-Gen and labels annotated by two human experts. Existence-canceling prompts have no silent or integrated example, since a target whose existence is cancelled should not appear in any status.

Borderline cases

The outcome labels are defined by the effect that the target’s appearance has on the image, and for some images this effect is a matter of degree. Figure 6 shows three pilot images on which the two annotators disagreed, one for each pair of labels that is hard to separate.

Office scene with a sponge

(a) prompt: The new employee absorbed every instruction like a sponge soaking up water. target: sponge labels: disruptive / silent

Department scene with a bowling ball

(b) prompt: The tiny disagreement rolled through the department like a bowling ball knocking down everything in its path. target: bowling ball labels: disruptive / integrated

Kitchen scene with a framed photograph of a cat

(c) prompt: The woman is making her breakfast, wondering what her cat is doing, since it is living with her parents right now. target: cat labels: silent / integrated

Figure 6. Borderline cases: (a) disruptive or silent, (b) disruptive or integrated, (c) integrated or silent. The two labels under each image are the two annotators’ labels.

Disruptive or silent. The two labels differ on whether the target’s appearance fits the scene. In Figure 6a, the sponge, the vehicle of the comparison, is drawn into the office: one annotator read it as an object an office may plausibly hold, the other as something that does not belong there.

Disruptive or integrated. The two labels differ on whether an exaggerated rendering is read as a stylized expression or as a scene that breaks common sense. In Figure 6b a bowling ball rolls through the department: one annotator took it as a stylization of the comparison, the other as an object that a real office cannot contain.

Integrated or silent. The two labels differ on whether a depiction of the target inside the scene is a form the prompt allows or the target slipping in. In Figure 6c the cat appears as a framed photograph on the counter: one annotator read the photograph as a fitting way to show an absent cat, the other as a cat that was not supposed to appear at all.

Unexpected over-realized targets

An annotator writes a prompt with one target in mind, but the generator may over-realize another mention, and even another family. Figure 7 shows two example images which over-realize both the vehicle of the comparison and the viewpoint. In (a) the leaves drift past the window (silent), and the student, who should face the lecturer, is turned toward the camera so that his face is visible (disruptive). In (b) a pot bubbles on the stove (silent), and the screen that carries the results faces the camera rather than the woman who is reading it (disruptive).

Two pilot images showing a student in a lecture and a woman in a kitchen

Figure 7. prompt: The student’s attention drifted like a leaf in the wind during the long afternoon lecture. targets: leaf (figurative), the student (perspectival) labels: silent / disruptive. prompt: Her excitement bubbled over like a pot on the stove when the results came in. targets: pot (figurative), the results (perspectival) labels: silent / disruptive. Two pilot images (Gemini) that over-realize two targets each: the vehicle of the comparison, silently, and the viewpoint of the person, disruptively. (a) The student faces the camera instead of the lecturer. (b) The screen faces the camera instead of the woman reading it.

In practical use, a generator is not told which mention to withhold, and a single prompt can over-realize several of them. Perspectival over-realization in particular can arise in any prompt that places a person or a text-bearing object in a scene, without a cue word. Methods that suppress a given word, such as a negative prompt or a reweighting of the attention on that word, must first be told which word to suppress, and a user does not say.

Dataset Details

Statistics

Table 3 gives the size of the raw and expanded splits of OverReal-Gen, the size of the gold and ambiguous splits of OverReal-Det, the label distribution of the gold split, and the agreement between the two annotators. When writing the three hand-written families, the annotators were asked not to reuse one template, so that the prompts cover diverse failures.

The ambiguous split

An image enters the ambiguous split when the label sets of its two annotators are disjoint (§ 4.2). Table 4 sorts the 588 such images, 20.7% of the 2,841 annotated images, by what the two annotators disagree on. In 245 of them, the annotators agree on whether the target is over-realized; in 266 of them, one annotator read the image as over-realized and the other as correct; the remaining 77 involve the label other.

Existence-cancelingMental-stateFigurativePerspectivalAll
Form of the outcome only65212562245
Over-realized against correct134314169266
One label is other417332377

Table 4. The ambiguous split of OverReal-Det by the kind of disagreement.

Stability of the rates under the ambiguous images. The headline rates (Figure 3, Table 7) come from the automatic annotator, so the ambiguous images do not enter them directly. To see how much they could matter, we recompute the over-realization rate on OverReal-Det with the ambiguous images included, taking each annotator’s labels in turn instead of the gold labels. The rate per family moves by at most 0.04 (existence-canceling 0.33 to 0.36 and 0.37, mental-state 0.71 to 0.73 and 0.72, figurative 0.65 to 0.65 and 0.64, perspectival 0.81 to 0.79 and 0.79), and the generator rankings stay almost unchanged (Spearman ρ≥0.77\rho\ge0.77 in every family).

Automatic Annotator

Cascade design

Rather than asking for one of four labels at once, the annotator follows the decision tree of Figure 8, one yes/no question per call. The first question is whether the target appears in the image. If it does not, the image is withheld when the rest of the image is a valid rendering of the prompt, and other when it is not. If it does, the next question is whether the target appears in the way the prompt calls for. If so, the same validity question decides between integrated and other; if not, the image is disruptive when the target’s appearance breaks the picture, and silent when it does not. The questions that instantiate these decisions differ by family for higher precision. The full question set per family is given in Appendix E.1.

Figure 8. The annotation cascade as a decision tree. Each decision node (blue) is one yes/no question put to the annotator in a separate call, with the image and the prompt; the leaves (italic) are the labels. The family-specific wording of the questions is given in Appendix E.1.

Self-evolution of the AI annotator

The AI annotator is adapted to the task without updating any model weights, by a form of self-evolution: the model annotates a development set with its current questions, revises the questions from the mistakes it made, and repeats until its agreement with the human labels stops improving. We build it on the human labels of OverReal-Det, split evenly, stratified by family and generator, into a development split and a test split; the development split alone drives the self-evolution, and the frozen annotator is then evaluated once on the test set. In each round the model scores the development split, receives the images on which its label disagrees with the human one, traces each disagreement to the question that produced it, and proposes a rewrite of that question; we accept the rewrite and score the revised questions again. The best wording is selected per family.

Agreement with the human labels

The annotator uses Claude Opus 5 [2] as the base model. Table 5 reports the AI-human agreement and human-human agreement in Cohen’s κ\kappa. Conclusions are given as follows.

DecisionExistence-cancelingMental-stateFigurativePerspectivalOverall
Four labelsAI-human1.000.420.600.410.58
Four labelshumans0.830.470.670.750.70
Target appearsAI-human1.000.740.820.710.83
Target appearshumans0.880.840.830.870.86
Over-realized vs. correctAI-human1.000.530.690.600.70
Over-realized vs. correcthumans0.880.490.740.790.75

Table 5. Cohen’s κ\kappa of the automatic annotator (Claude Opus 5) against the human gold labels on the held-out test split of OverReal-Det (1,118 images), and of the two human annotators against each other on the 2,688 doubly annotated images with one label each, for the four-outcome label, for whether the target appears, and for the collapse to over-realized against correct.

Self-evolution improves the cascade annotator. Asking the model for one of the four labels outright is unreliable: a multiple-choice prompt over the four labels reached only κ=0.46\kappa= 0.46 against the human labels. Self-evolution of the cascade questions then raised the agreement with the human labels on the development split from κ=0.61\kappa= 0.61 to 0.68 over the four labels, and from 0.65 to 0.76 on the split between over-realization and correct output.

The annotator agrees with the human labels nearly as well as the humans agree with each other. Over the four labels, κ\kappa is 0.58 overall against 0.70 between the humans; on the split between over-realization and correct output, which the rates of Table 7 rest on, it is 0.70 against 0.75, a substantial agreement. The automatic labels are therefore reliable enough for the rates we report, if not for a fine reading of every image.

The agreement holds on prompts the annotator never saw during development. The development and test splits of OverReal-Det were drawn over images, so 87% of the 1,118 test images share a prompt with a development image. On the 148 test images whose prompt never appears in the development split, κ\kappa is 0.54 over the four labels and 0.66 on the split between over-realization and correct output, against 0.58 and 0.70 on the full test split.

Part of the gap to the humans is the ambiguity of the task. The κ\kappa values in Table 5 are not uniformly high, but they come close to the agreement between the humans on this task. Deciding whether the target appears is easy for both the annotator and the humans (κ=0.83\kappa= 0.83 against 0.86). Deciding what its appearance amounts to is harder for both, because the outcome labels are defined by the effect that the target’s appearance has on the image, and this effect is a matter of degree, as described in the limitations; on mental-state prompts the annotator is close to the humans (0.42 against 0.47). On perspectival prompts it falls short of them (0.41 against 0.75), so there the gap is the model’s. The label with the least sharp boundary is integrated (Appendix § 12.§ 2): on whether an image is integrated, the two humans reach κ=0.52\kappa= 0.52.

Results

Conventional Evaluation: LLM Judge Details

Both judges score the same images with the verbatim VIEScore text-to-image semantic-consistency template: Claude Opus 5 (claude-opus-5, accessed 2026-09) and GPT-5.2 (gpt-5.2-2025-12-11). Table 6 reports the Opus rating; Table 8 gives the GPT-5.2 rating for the same cells and Table 9 the agreement of the two judges.

Existence-cancelingMental-stateFigurativePerspectival
GPT-Image9.58.89.26.3
Nano Banana9.18.89.06.2
Ideogram9.07.88.25.2
FLUX.1-dev6.27.77.94.9
Qwen-Image5.08.38.75.3
OmniGen26.07.68.05.0
SD3.5-Medium7.17.47.45.0
SD3.5-Large4.97.68.15.0

Table 8. LLM judge (LJ) ratings by GPT-5.2, the same cells as the LJ column of Table 6.

FamilyOpus meanGPT meanPearson rrMean ∣Δ∣|\Delta|nn
Existence-canceling6.86±2.836.86 \pm2.837.37±3.277.37 \pm3.270.901.021700
Mental-state6.78±1.526.78 \pm1.527.97±1.307.97 \pm1.300.621.321750
Figurative6.71±1.956.71 \pm1.958.29±1.398.29 \pm1.390.571.691800
Perspectival3.63±1.813.63 \pm1.815.27±2.385.27 \pm2.380.561.981800
Overall5.98±2.495.98 \pm2.497.22±2.527.22 \pm2.520.781.517050

Table 9. Between-judge statistics: mean rating ±\pm standard deviation per judge, correlation, and mean absolute rating difference.

Intervals for the over-realization rates

Table 10 gives the over-realization rate of Table 7, the sum of the DO and SO columns, with half the width of a 95% bootstrap interval.

Existence-cancelingMental-stateFigurativePerspectivalAll families
GPT-Image.02 ± .03.61 ± .12.51 ± .11.69 ± .10.43 ± .05
Nano Banana.12 ± .09.56 ± .11.41 ± .10.95 ± .04.49 ± .05
Ideogram.08 ± .07.53 ± .11.32 ± .09.83 ± .08.41 ± .04
FLUX.1-dev.43 ± .13.73 ± .11.39 ± .10.75 ± .10.53 ± .06
Qwen-Image.64 ± .13.89 ± .07.62 ± .10.89 ± .06.73 ± .05
OmniGen2.51 ± .11.86 ± .08.45 ± .11.81 ± .08.61 ± .05
SD3.5-Medium.34 ± .11.87 ± .07.40 ± .10.78 ± .09.54 ± .05
SD3.5-Large.64 ± .12.84 ± .08.61 ± .10.76 ± .10.68 ± .06

Table 10. Over-realization rate of each system per family and over all families, weighting the families by their prompt counts in OverReal-Gen, ± half the width of a 95% bootstrap interval over prompts.

Outcomes across the two seeds

For the raw prompts of OverReal-Gen, each model generates two images per prompt under different seeds. Table 11 conditions on the label of one image and gives the distribution of the label of the other, over the labeled sample of §5.3 (172 silent, 428 correct and 588 disruptive conditioning images). Existence-canceling prompts have no silent images.

Existence-cancelingMental-stateFigurativePerspectivalAll
P(DO1∣DO0)P(\mathrm{DO}_{1} \mid\mathrm{DO}_{0}).84.77.65.71.79
P(SO1∣DO0)P(\mathrm{SO}_{1} \mid\mathrm{DO}_{0}).00.17.10.20.10
P(C1∣DO0)P(\mathrm{C}_{1} \mid\mathrm{DO}_{0}).16.06.26.09.12
P(DO1∣SO0)P(\mathrm{DO}_{1} \mid\mathrm{SO}_{0})—.44.05.64.33
P(SO1∣SO0)P(\mathrm{SO}_{1} \mid\mathrm{SO}_{0})—.50.76.32.57
P(C1∣SO0)P(\mathrm{C}_{1} \mid\mathrm{SO}_{0})—.06.19.04.10
P(DO1∣C0)P(\mathrm{DO}_{1} \mid\mathrm{C}_{0}).16.23.09.19.16
P(SO1∣C0)P(\mathrm{SO}_{1} \mid\mathrm{C}_{0}).00.09.13.03.04
P(C1∣C0)P(\mathrm{C}_{1} \mid\mathrm{C}_{0}).84.68.78.78.80

Table 11. Label of the other seed’s image given the label of one image of the same prompt and open model, on the raw prompts of OverReal-Gen. P(y1∣y0)P(y_{1} \mid y_{0}) is the share of images with label y0y_{0} whose other-seed image has label y1y_{1}; DO disruptive, SO silent, C correct (integrated or withheld). Existence-canceling prompts have no silent images.

Conventional metrics on matched pairs

For the raw prompts of OverReal-Gen, each model generates two images per prompt under different seeds. Over the labeled sample of § 5.3, we take the pairs whose two images differ in outcome, one over-realized and one correct, and ask which of the two each conventional metric scores higher. Table 12 gives this winning rate of the over-realized image, by kind of over-realization and per family.

MetricWinning rateDisruptiveSilentExistence-cancelingMental-stateFigurativePerspectival
CLIPScore.64 ± .10.69.44.75.56.60.38
VQAScore.33 ± .13.25.80.05.67.75.50
PickScore.66 ± .10.72.44.72.67.60.50
HPSv2.65 ± .11.69.50.65.72.70.38
LLM judge.12 ± .08.06.50.00.27.33.17

Table 12. Winning rate: share of matched pairs (same prompt, same open model, two seeds; one image over-realized, one correct) in which the metric scores the over-realized image higher, ± half the width of a 95% bootstrap interval over prompts.

The conventional metrics reward over-realization or are indifferent to it. CLIPScore, PickScore and HPSv2 score the over-realized image higher in 64% to 66% of the pairs. VQAScore is split: on existence-canceling prompts it scores the correct image higher in 95% of the pairs, but on mental-state and figurative prompts it scores the over-realized image higher in 67% and 75%, and on perspectival prompts it scores either image higher equally often (50%). The LLM judge scores the correct image higher in 88%. The preference of the three metrics is strongest on the disruptive images (69% to 72%); on the pairs whose over-realized image is silent they score either image higher about equally often (44% to 50%), so they do not penalize the silent target either. A metric that credited correct realization would score the correct image higher in both cases; the conventional metrics reward the disruptive over-realization and are indifferent to the silent one.

Prompt expansion: full results

Table 13 gives the outcome distribution of the five open generators on the raw prompts and on the prompts rewritten by the Qwen-Image rewriter and by Ideogram Magic Prompt, per family, with the change from the raw prompt in parentheses. § 6.2 discusses the over-realization rates; Appendix § 15.§ 7 classifies what each rewrite does with the target.

Experiment on the in-the-wild set

The curated prompts of OverReal-Gen are written or synthesized for the benchmark, and the hand-written families are admitted only after a pilot model over-realized them (§ 4.1). We also evaluate the systems on an in-the-wild set of prompts that users wrote for their own purposes, selected on the wording of the prompt alone.

Sources and sampling We pool the prompts of DiffusionDB (Wang et al. [34]) and Pick-a-Pic (Kirstain et al. [20]), collapse whitespace, drop duplicates, drop DiffusionDB prompts whose text or image carries an NSFW score of 0.5 or more, keep prompts of 3 to 120 words that are at least 90% ASCII, and draw 7,000 DiffusionDB and 3,000 Pick-a-Pic prompts at random.

Classification Each of the 10,000 prompts is classified into one of the four families of § 3.1 or none, based on the cue words of each family. Of the 10,000 prompts, 202 (2.0%) qualify: 62 existence-canceling, 3 mental-state, 129 figurative and 8 perspectival. To obtain enough prompts of the two rare families, we add another 135 mental-state and 60 perspectival prompts from the rest of the datasets, also based on cue words.

Experiments The 397 prompts are generated under the settings of § 5, one image per prompt and system, and the images are labeled by the automatic annotator of § 5.3. Table 7 gives the results, and the conclusions of § 6.1 carry over. First, every system over-realizes in every family: pooled over all systems, the over-realization rate is 0.60 on existence-canceling prompts, 0.28 on figurative, 0.51 on mental-state and 0.80 on perspectival. Second, the form of the failure follows the family: perspectival targets break the scene (mostly DO), figurative targets slip in silently (almost entirely SO), and mental-state targets sit between; the proprietary systems handle mental-state prompts largely by integrated depiction (I of 0.58 to 0.64), which the open models rarely manage. Third, the curated prompts run 0.1 to 0.3 higher than the in-the-wild prompts in the mental-state and figurative families.

Existence-canceling goes the other way: negations in the wild mostly cancel an attribute of the subject (no nose, faceless, without laces) rather than an object in the scene, and every system, the proprietary ones included, over-realizes at least 28% of them, far above its rate on the templated prompts of § 4.1.

Model rankings on the two sets Ranking the eight systems by their over-realization rate, the Spearman correlation between the two sets is 0.90 on existence-canceling and 0.83 on mental-state prompts, so the rankings there are stable. On figurative and perspectival prompts it is near zero (−0.26 and 0.05), but there the systems lie close together on either set: in the wild, all eight fall between 22% and 32% on figurative and between 74% and 88% on perspectival prompts, so the ranking itself carries little information. The form of the failure also agrees: Table 14 gives the disruptive share of over-realization, pooled over systems, and the two sets agree within five points except in the figurative family. That gap has two causes. (1) By a keyword rule checked by hand, about 90% of the wild figurative prompts compare appearance or shape (a cloud shaped like a snake) and 10% compare a quality or behavior (as stubborn as a mule); on OverReal-Gen the split is about 15% against 85%. In an appearance comparison the target’s look is part of the request, so rendering it rarely breaks the scene; in a quality comparison the target has no place in the scene, so rendering it does. (2) The annotators of OverReal-Gen were asked not to reuse one template, so that the prompts cover diverse failures, whereas the wild figurative prompts are dominated by “X shaped like Y” art prompts.

OverReal-GenIn the wild
Existence-canceling1.001.00
Mental-state.69.66
Figurative.35.08
Perspectival.79.84

Table 14. Disruptive share of over-realization, DO/(DO+SO), pooled over the eight systems, on the raw prompts of OverReal-Gen and on the in-the-wild prompts.

Nano Banana and the screening Since the hand-written prompts were screened with Gemini (§4.1), we check whether the screening sets Nano Banana apart from the other systems. Ranked by over-realization rate from highest to lowest, Nano Banana is 6th, 7th, 5th and 1st on OverReal-Gen (existence-canceling, mental-state, figurative, perspectival) and 7th, 7th, 7th and 6th in the wild. The screening changes its rank only on perspectival prompts; on the other three families its rank is the same on the two sets.

Human check of the automatic labels The in-the-wild rates rest on the automatic annotator alone, whose agreement with the humans was measured on the curated images of OverReal-Det (Table 5). To check its labels on the wild images, we drew a stratified sample of 202 of them, stratified by family and automatic label with the perspectival family oversampled (100 images, and 34 from each other family), and one of the authors relabeled them by hand, blind to the automatic label. Table 15 gives Cohen’s κ\kappa between the two, with the values on OverReal-Det for reference, and the over-realization rate after reweighting: each automatic label is replaced by the distribution of human labels observed for it in the sample. The automatic labels hold up. Over the four labels, the agreement is 0.70 in the wild, the same as the two human annotators’ agreement with each other on OverReal-Det, and 0.65 on the perspectival images, close to the human-human value of 0.75. The annotator over-calls over-realization slightly: the reweighted rate is below the automatic one in every family, by 2 to 12 points, but the pattern across families is unchanged, and perspectival prompts remain far above the others at 68%.

Familynn\multicolumn{2}{c}{κ\kappa, four labels}\multicolumn{2}{c}{κ\kappa, OR vs. correct}\multicolumn{2}{c}{OR rate in the wild}
wildDet, AI-humanDet, humanswildautomaticreweighted
Existence-canceling34.821.00.83.82.60.51
Mental-state34.76.42.47.76.51.42
Figurative34.53.60.67.53.28.26
Perspectival100.65.41.75.69.80.68
All202.70.58.70.72——

Table 15. Human check of the automatic labels in the wild. Cohen’s κ\kappa between the automatic annotator and a human annotator on a stratified sample of nn wild images, over the four labels and over the collapse to over-realized (OR) against correct, with the AI-human and human-human values on OverReal-Det (Table 5) for reference; and the over-realization rate in the wild from the automatic labels and after reweighting them by the human labels.

Conclusions

First, apart from existence-canceling, the systems behave on the prompts that users wrote much as they do on the prompts we wrote: the pattern across families and systems is the same, and the rates differ by at most 0.3. The curated prompts are harder, as intended, but they do not produce a failure that natural prompts lack, so the results of §6 are not an artifact of the curation. Second, the four situations account for about 2% of the prompts in current text-to-image datasets, so existing benchmarks under-represent them and have not evaluated them. Mental-state and perspectival prompts are the rarest, under 0.1% each. We keep them in OverReal nonetheless: the same situations arise whenever a prompt describes what a character thinks or what a viewpoint hides, which is routine in narrative prompts and in video generation. A model that cannot keep such a target out of a single image will not keep it out of a video either.

How the rewriters treat the target

For the analysis of § 6.2 we read every rewritten prompt and recorded what became of the target. Each rewrite falls into one of four categories:

  • dropped: the target is no longer mentioned, under any name;

  • explicitly excluded: the rewrite states that the target is absent or must not appear (“no keyboard is present”, “the letter p is obscured by shadow”);

  • mentioned: the target is named but nothing in the rewrite asks for it to be shown, whether it keeps the status the original prompt gave it (“remembering the trumpet”, “as fierce as a tiger”, “inside the closed box is a teapot”) or not;

  • explicitly depicted: the rewrite describes the target as something to render in view (“a tiger stands in the corner”, “visible through a gap in the lid”, “the hidden side reveals a red tower”).

A first pass assigned the categories by rule, matching the target’s words against the rewrite and looking for negation and visibility phrases near them; every rewrite that the rules flagged, every rewrite marked as dropped that still produced an over-realized image, and a random sample of the rest were then read and corrected by hand. Eight existence-canceling items whose target is a normal part of the scene, such as a keyboard beside a laptop, are left out of this analysis. Table 16 gives the distribution per family and rewriter, and Table 17 the over-realization rate of the corresponding images by category.

FamilyRewriterDroppedExcludedMentionedDepicted
Existence-cancelingQwen09820
Ideogram95220
Mental-stateQwen016435
Ideogram4104019
FigurativeQwen150778
Ideogram550369
PerspectivalQwen014256
Ideogram1024445

Table 16. How the rewriters treat the target, as a percentage of the rewritten prompts of each family: the target is dropped, explicitly excluded (“no keyboard is present”), mentioned, or explicitly to be depicted (“visible through a gap”).

FamilyDroppedExcludedMentionedDepicted
Existence-canceling0355—
Mental-state4—7182
Figurative4—3375
Perspectival21008687
Overall2386084

Table 17. Over-realization rate (percent) of the images generated from rewritten prompts, by how the rewrite treats the target, pooled over both rewriters and the five open generators. Cells with fewer than 10 images are left blank.

Text encoder

The backbone never sees the prompt, only the encoder’s output, so we ask whether that output still distinguishes the original prompt from one that calls for the target. Both experiments below use the plain-mention controls of § 6.3: in a control, the words that keep the target out of the scene are replaced by words that put it in, and the target word is kept, so that “The coach is as fierce as a tiger” becomes “The coach is standing next to a tiger” (one example per family in Table 20).

Over-realization is not due to a lack of information in the encoder output, except on perspectival prompts. A linear probe is trained to tell the two prompts apart from the contextual vector of the target word, at the layer the pipeline feeds to the backbone, for the four encoders of the open models: CLIP-L and CLIP-bigG [29] (SD3.5), T5-XXL [30] (SD3.5 and FLUX) and Qwen2.5-VL-7B [3] (Qwen-Image). It is trained on all four families together and tested with cross-validation grouped by target word. On existence-canceling, mental-state and figurative prompts the probe separates the two prompts at 81% to 100%, while on perspectival prompts it stays close to chance, at 53% to 62%, at every layer (Table 18, Figure 9). Furthermore, within each of the three families, the Spearman correlation between an item’s probe margin and its over-realization rate lies between −0.28 and 0.28, with no consistent sign across encoders. Thus, for existence-canceling, mental-state and figurative prompts, over-realization is not explained by the encoding process; on perspectival prompts the probe finds no linear evidence in the encoder output that the target is out of view.

EncoderExistence-cancelingMental-stateFigurativePerspectivalAll
CLIP-L/14.95 ± .04.81 ± .06.87 ± .03.53 ± .02.79 ± .02
CLIP-bigG/14.96 ± .03.87 ± .05.87 ± .03.60 ± .03.82 ± .02
T5-XXL.95 ± .04.89 ± .04.94 ± .02.62 ± .03.85 ± .02
Qwen2.5-VL-7B1.00 ± .00.92 ± .04.91 ± .02.60 ± .03.86 ± .01

Table 18. Accuracy of a linear probe that tells an original prompt from its plain-mention control, from the contextual vector of the target word at the layer each pipeline feeds to the backbone, ±\pm half the width of the bootstrap 95% interval. Chance is 50%.

Figure 9. Accuracy of the target-word probe of Table 18 at every layer of each encoder, per family and pooled. Layer 0 is the embedding output; the dashed line marks the layer the pipeline feeds to the backbone. The perspectival family stays near chance at every layer of every encoder.

Qwen-Image’s encoder knows the target should not be visible, and the generator shows it anyway. Qwen-Image’s encoder is also an instruction model, so we ask it directly: “If this prompt were rendered as an image exactly as described, would a ⟨target⟩ be visible in the image? Answer yes or no.”, expecting no for the original prompt and yes for its control. It answers correctly on 100%, 75% and 84% of the existence-canceling, mental-state and figurative prompts and on 56% of the perspectival ones (Table 19); the same weights then condition images in which the target appears in more than half of the cases (Table 7). This is another piece of evidence that the information survives the encoding process for existence-canceling, mental-state and figurative prompts; for the perspectival family we find no evidence that it does.

FamilyOriginal prompt (no)Plain-mention control (yes)Both
Existence-canceling1.00.76.88
Mental-state.751.00.87
Figurative.84.93.89
Perspectival.56.85.70

Table 19. Accuracy of Qwen2.5-VL-7B, the text encoder of Qwen-Image, when asked directly whether the target would be visible.

Backbone attention: per family

Table 21 gives the attention shares of § 6.3 per family. Pairs per family: 100, 150, 472 and 368 for existence-canceling, mental-state, figurative and perspectival prompts; every value pools the two seeds.

The cue words lower the target’s attention only slightly, and not at all on perspectival prompts. On existence-canceling, mental-state and figurative prompts the target receives less attention under the original prompt than under the control in every model (ρt\rho_t 0.74 to 0.92). On perspectival prompts ρt\rho_t is 0.97 to 0.99: the cue words make almost no difference, consistent with Appendix § 15.8, where the perspectival information is already lost in the encoder.

Backbone attention under prompt expansion

Expanded prompts are about five times longer than the original ones, so for them we report the relative attention rr: each token’s share divided by the prompt’s average share. Table 22 gives the target’s rr under the original prompt and under the two expansions of § 6.2, over the expanded prompts in which the target survives verbatim (both seeds).

Familynn Qwennn Ideo.FLUX.1-dev orig.FLUX.1-dev QwenFLUX.1-dev Ideo.SD3.5-Large orig.SD3.5-Large QwenSD3.5-Large Ideo.Qwen-Image orig.Qwen-Image QwenQwen-Image Ideo.
Existence-canceling1002.771.05—.85.99—.82.58—
Mental-state13676.961.601.851.131.051.42.811.321.05
Figurative3981961.001.371.351.401.091.14.781.431.17
Perspectival282232.871.301.15.941.001.06.741.141.01
All families916506.901.331.451.081.041.21.791.121.08

Table 22. Relative attention rr of the target words under the original prompt (orig.) and under the expanded prompts of the Qwen rewriter and of Ideogram Magic Prompt (Ideo.), per family and over all families; nn is the number of expanded images per family and rewriter. Ideogram Magic Prompt keeps the target in only two existence-canceling prompts, so that cell is left out and the Ideogram mean excludes the family.

Expansion does not lower the target’s attention. Under the expanded prompts in which the rewriter kept the target, the target receives at least as much attention as an average word in every model and for both rewriters (rr = 1.04 to 1.45). The one exception is Qwen-Image on existence-canceling prompts under the Qwen rewriter (0.58): the rewriter restates the cancellation as a clause of its own, such as No toilet is visible in the frame, and inside that clause Qwen-Image gives the target about half the attention of an ordinary word, while FLUX.1-dev and SD3.5-Large read it like any other word (1.05 and 0.99). Figure 10 shows two items under the original prompt, the control and the two rewrites.

Explicit constraint in the prompt

This experiment tests whether a generator withholds the target when the prompt says so explicitly. We append the sentence “No {target} should appear in the image.” to all 75 mental-state prompts and to a random half (118) of the figurative prompts of OverReal-Gen, and generate one image per prompt (seed 0) with GPT-Image-1.5 and the five open-weight generators. The automatic annotator labels these images against the original prompt, so the labels keep their usual meaning. As the baseline we take the images of the same prompts in the labeled sample of § 5.3 (raw prompts, both seeds).

FamilyOriginal promptPlain-mention control
Existence-cancelinga cat sitting on a couch with a dog on the floor, with no book in sighta cat sitting on a couch with a dog on the floor, with a book in sight
Mental-stateTim sits alone at a classroom desk, remembering the toy robot he lost years ago.Tim sits alone at a classroom desk, holding the toy robot he got years ago.
FigurativeThe coach is as fierce as a tiger.The coach is standing next to a tiger.
PerspectivalA desk drawer is completely closed. Inside is a green frog toy.A desk drawer is pulled open. Inside is a green frog toy.

GPT-Image declined eight of the constrained prompts (five mental-state, three figurative), which are left out. Table 23 gives the outcome distribution per generator, and pooled over the five open-weight generators.

Generator + promptMental-state nnTargetDOSOIWFigurative nnTargetDOSOIW
GPT-Image-1.5 (raw)100.89.35.26.28.1142.57.14.26.17.43
+ constraint70.14.01.03.10.86115.08.00.05.03.92
FLUX.1-dev (raw)99.76.44.28.03.2447.47.09.23.15.53
+ constraint74.77.50.23.04.23116.59.15.37.07.41
Qwen-Image (raw)99.90.67.22.01.1045.69.27.38.04.31
+ constraint75.91.72.19.00.09118.76.26.43.07.24
OmniGen2 (raw)100.89.64.22.03.1151.51.12.31.08.49
+ constraint75.83.60.20.03.17114.61.22.32.07.39
SD3.5-Medium (raw)100.88.66.21.01.1260.40.08.25.07.60
+ constraint75.87.44.43.00.13116.59.17.35.06.41
SD3.5-Large (raw)100.87.61.23.03.1353.68.17.38.13.32
+ constraint75.87.68.19.00.13117.74.25.39.10.26
Open-weight, pooled (raw)498.86.60.23.02.14256.54.14.31.09.46
+ constraint374.85.59.25.01.15581.66.21.37.07.34

Table 23. Outcome distribution on the raw prompts and on the same prompts with the explicit constraint “No {target} should appear in the image.” appended, for the mental-state prompts and a random half of the figurative prompts of OverReal-Gen. Target is the share of images in which the target appears (DO + SO + I); DO disruptive, SO silent, I integrated, W withheld. The raw rows are over the labeled sample of §5.3 (both seeds), the constraint rows over one image per prompt (seed 0).

GPT-Image follows the constraint; the open-weight generators do not. With the constraint, the target almost disappears from GPT-Image’s output: it appears in 14% of the mental-state images instead of 89%, and in 8% of the figurative images instead of 57%. For the five open-weight generators the target appears as often as before in the mental-state family (85% against 86%) and more often in the figurative family (66% against 54%). This matches their rates on the existence-canceling prompts, which are explicit negations by construction: the open-weight generators over-realize 34% to 64% of them (Table 7). So GPT-Image is able to exclude the target when told to. On the original prompt it tends to treat the mentioned target almost as a requested one and renders it, unless the prompt rules it out explicitly.

An explicit constraint is not a general fix. A prompt contains several mentions, any of which the model may over-realize (Appendix § 12.§ 3), and in practice a user does not say which mention to withhold. The fix should come from a model that infers which mentions belong in the scene, not from a prompt that spells it out.

Full Prompts and Annotation Guidelines

Automatic annotation protocol

Each image is scored by a cascade of yes/no questions, one API call per question, with the image, the question and, from the second question on, the original prompt. T stands for the target; the article is adjusted to the target. Every family starts with the same screening question, “Is this image severely corrupted, or unrelated to the scene the prompt describes?”, whose yes-answer gives other.

Existence-canceling Q1: Is there a TT anywhere in this image? (yes: disruptive; no: withheld)

Mental-state Q1: Is there a TT anywhere in this image? (no: withheld) Q2: In this image, is a TT physically present in the scene, as an actual thing that is there with the person? (no: Q3) Q2b: In this image, is the TT right beside the person and prominent in the picture, as opposed to distant, small or in the background? (yes: disruptive; no: silent) Q3: In this image, is a TT shown as something the person imagines, remembers or believes, for example inside a thought bubble, a dream haze, a separate inset or a ghostly overlay? (yes: integrated; no: silent)

Figurative Q1: Is there a TT anywhere in this image? (no: withheld) Q2: In this image, does the presence of a TT make the scene absurd, impossible or nonsensical as a picture of what the prompt describes? (yes: disruptive) Q3: In this image, is a TT merged with the person or thing it describes, as a deliberate visual metaphor in a stylized or poster-like composition? (yes: integrated; no: silent)

Perspectival, hidden object Q1: Is there a TT anywhere in this image, in any form, including as a print, picture, sticker, logo or drawing? (no: withheld) Q2: According to the prompt, the TT should be hidden from the viewer: inside, behind, underneath, or on the far side of something. In this image, is the TT nevertheless exposed to the viewer at that very spot, for example because the container, door, curtain, barrier or surface is shown open, transparent, removed, or turned the wrong way? (yes: disruptive) Q3: In this image, is the T shown only indirectly, through a reflection, an inset, a thought bubble, or a picture taken from the in-scene person’s own point of view? (yes: integrated; no: silent)

Perspectival, hidden text Q1: In this image, is the written side of the item that carries “T” (the letter, note, page, screen, card or sign the prompt describes) turned toward the viewer, so that its writing can be seen, whether or not it is legible? (no: withheld) Q2: In this image, is the writing “T” shown to the viewer only through a reflection, an inset, a thought bubble, or a picture taken from the in-scene person’s own point of view? (yes: integrated) Q3: According to the prompt, the writing “T” should be out of the viewer’s sight. In this image, is it nevertheless in plain view, on the very item the prompt describes? (yes: disruptive; no: silent)

Annotation guide

Annotators labeled through a web interface that showed the image, the prompt, and the target, together with the guide below (reproduced verbatim); each person could pick one or two of the five labels per image.

  • disruptive: the target appears, and its appearance causes at least one of: (1) the image breaks common sense or logic, e.g. a reader facing the back of the letter they are reading, a screen facing the camera instead of its user, a tiger on a sports field; (2) the image contradicts the prompt, e.g. the prompt says the target is absent, or sealed inside an opaque box, yet it is visible.

  • silent: the target appears; unexpected, but not disruptive, typically as background or a small detail. E.g. “her face is like a red apple” shown as a red-faced child eating an apple; “the roof is like a banana” with banana trees in the background.

  • integrated: the target appears, but in a natural way, within what a normal viewer would accept. E.g. “the roof is like a banana” as a banana-shaped roof and no banana trees; a thought bubble depicting what a character imagines; contents shown inside a closed drawer as such; a poster style that merges a person with their attribute.

  • withheld: the target does not appear, and the image does not contradict the prompt. E.g. something in a closed bag shown only as the closed bag; “her face is like a red apple” as just a red-faced child.

  • other: the image is severely corrupted, or does not match the prompt at all.

You may select two labels, but please prefer a single label; pick two only when the case is genuinely ambiguous.

References

  1. [1]Kumail Alhamoud, Shaden Alshammari, Yonglong Tian, Guohao Li, Philip H. S. Torr, Yoon Kim, and Marzyeh Ghassemi. Vision-language models do not understand negation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.arxiv.org/abs/2501.09425
  2. [2]Anthropic. Claude Opus 5. https://docs.anthropic.com/en/docs/about-claude/models, 2026. Model version claude-opus-5, accessed September 2026.
  3. [3]Shuai Bai, Keqin Chen, Xuejing Liu, et al. Qwen2.5-VL technical report. arXiv preprint arXiv:2502.13923, 2025.arxiv.org/abs/2502.13923
  4. [4]James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, Wesam Manassra, Prafulla Dhariwal, Casey Chu, Yunxin Jiao, and Aditya Ramesh. Improving image generation with better captions. Technical report, OpenAI, 2023.
  5. [5]Black Forest Labs. Flux.1. https://github.com/black-forest-labs/flux, 2024.
  6. [6]Yuliang Cai, Jesse Thomason, and Mohammad Rostami. TNG-CLIP: Training-time negation data generation for negation awareness of CLIP. arXiv preprint arXiv:2505.18434, 2025.arxiv.org/abs/2505.18434
  7. [9]Colin Conwell, Rupert Tawiah-Quashie, and Tomer Ullman. Relations, negations, and numbers: Looking for logic in generative text-to-image models. arXiv preprint arXiv:2411.17066, 2024.arxiv.org/abs/2411.17066
  8. [10]Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In International Conference on Machine Learning, 2024.arxiv.org/abs/2403.03206
  9. [11]Gilles Fauconnier. Mental Spaces: Aspects of Meaning Construction in Natural Language. Cambridge University Press, 1994.
  10. [12]Google DeepMind. Gemini 2.5 flash image. https://developers.googleblog.com/en/introducing-gemini-2-5-flash-image/, 2025.
  11. [13]Yaru Hao, Zewen Chi, Li Dong, and Furu Wei. Optimizing prompts for text-to-image generation. In Advances in Neural Information Processing Systems, 2023.arxiv.org/abs/2212.09611
  12. [14]Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A reference-free evaluation metric for image captioning. In Proceedings of EMNLP, 2021.
  13. [15]Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A. Smith. TIFA: Accurate and interpretable text-to-image faithfulness evaluation with question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.arxiv.org/abs/2303.11897
  14. [16]Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2I-CompBench: A comprehensive benchmark for open-world compositional text-to-image generation. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2023.DOI
  15. [17]Ideogram. Magic prompt. https://docs.ideogram.ai/using-ideogram/generation-settings/magic-prompt, 2024.
  16. [18]Ideogram. Ideogram 3.0. https://about.ideogram.ai/3.0, 2025.
  17. [20]Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. In Advances in Neural Information Processing Systems, 2023.arxiv.org/abs/2305.01569
  18. [21]Max Ku, Dongfu Jiang, Cong Wei, Xiang Yue, and Wenhu Chen. VIEScore: Towards explainable metrics for conditional image synthesis evaluation. In Proceedings of ACL, 2024.
  19. [22]Youngsun Lim, Hojun Choi, and Hyunjung Shim. Evaluating image hallucination in text-to-image generation with question-answering. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2025. Preprint. Under review.
  20. [23]Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. In European Conference on Computer Vision, 2024.arxiv.org/abs/2404.01291
  21. [24]OpenAI. GPT-5.2. https://platform.openai.com/docs/models/gpt-5.2, 2025a. Model version gpt-5.2-2025-12-11.
  22. [25]OpenAI. GPT image in the API. https://platform.openai.com/docs/models/gpt-image-1.5, 2025b.
  23. [26]Junsung Park, Jungbeom Lee, Jongyoon Song, Sangwon Yu, Dahuim Jung, and Sungho Yoon. Know “no” better: A data-driven approach for enhancing negation awareness in CLIP. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2825–2835, 2025.arxiv.org/abs/2501.10913
  24. [29]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning (ICML), 2021.arxiv.org/abs/2103.00020
  25. [30]Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67, 2020.arxiv.org/abs/1910.10683
  26. [31]Kaiyue Sun, Rongyao Fang, Chengqi Duan, Xian Liu, and Xihui Liu. T2I-ReasonBench: Benchmarking reasoning-informed text-to-image generation. arXiv preprint arXiv:2508.17472, 2025.arxiv.org/abs/2508.17472
  27. [33]Zengbin Wang, Xuecai Hu, Yong Wang, Feng Xiong, Man Zhang, and Xiangxiang Chu. Everything in its place: Benchmarking spatial intelligence of text-to-image models. In International Conference on Learning Representations (ICLR), 2026.arxiv.org/abs/2601.20354
  28. [34]Zijie J. Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duan Horng Chau. DiffusionDB: A large-scale prompt gallery dataset for text-to-image generative models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, 2023.arxiv.org/abs/2210.14896
  29. [35]Chenyuаn Wu, Pengfei Zheng, Ruiran Yan, Shitao Xiao, et al. OmniGen2: Exploration to advanced multimodal generation. arXiv preprint arXiv:2506.18871, 2025.
  30. [36]Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.09341, 2023.arxiv.org/abs/2306.09341

Paper details

vlnc:W_4zqy36u5
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 5

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Claude Code

View statement
Looking for related papers…
Contents