WorksComputer Science
KnowVis: Dual-View Evaluation and Diagnosis of World-Knowledge Grounding in Text-to-Image Models
Abstract
Text-to-image (T2I) models have made substantial progress in visual realism, aesthetic quality, and instruction following. However, real-world prompts often go beyond explicit visual descriptions and require implicit facts, structured knowledge, and domain-specific commonsense. Existing evaluations mainly focus on explicit prompt-to-image semantic alignment or specific knowledge subdomains, lacking unified evaluation and diagnosis of world-knowledge grounding in T2I models. We introduce KnowVis for dual-view evaluation and diagnosis of world-knowledge grounding in T2I generation. Built on a hierarchical knowledge taxonomy with case-level annotations, KnowVis combines two complementary benchmark views with a structured MLLM-based VQA judge protocol. The Skill-Tree Benchmark supports fine-grained atomic knowledge evaluation, while the Generalized Benchmark examines open-ended knowledge composition and expression. The protocol uses Required VQAs to assess atomic knowledge correctness in the Skill-Tree Benchmark, and combines Required, Optional, and dynamically discovered knowledge expressions with human-judge collaborative evaluation in the Generalized Benchmark. We validate the protocol's reliability on the Skill-Tree validation split, enabling scalable evaluation and diagnostic error attribution. Experiments show that KnowVis reveals notable gaps in world-knowledge grounding across existing T2I models and provides structured diagnostic signals for knowledge-intensive image generation.
Introduction
Text-to-image (T2I) generation models have made substantial progress in visual realism, aesthetic quality, and instruction following. Early systems such as DALL·E 2, Imagen, and Stable Diffusion established the modern T2I paradigm [1, 2, 3], while recent multimodal image generation systems and image-capable foundation models have further improved complex prompt following, text rendering, and practical visual content generation [4, 5, 6]. Despite these advances, many widely used evaluation protocols still primarily assess whether generated images correctly render the objects, attributes, and relations explicitly specified in the prompt [7, 8, 9].
However, real-world prompts often require more than surface-level prompt rendering. A prompt may implicitly require factual knowledge, physical commonsense, symbolic conventions, domain rules, cultural knowledge, or reasoning over unstated constraints. In such cases, the challenge is not only to “draw what is mentioned,” but also to “draw it correctly”: the generated image should satisfy the relevant world-knowledge constraints required by the prompt. Recent studies have shown that T2I models can still fail on such knowledge-intensive tasks, including physical commonsense, factual concept generation, and beyond-prompt semantic consistency [10, 11, 12, 13, 14, 15]. These failures suggest that, in knowledge-grounded generation, errors cannot always be fully explained by generic image quality or surface-level text-image alignment alone. We refer to this capability as world-knowledge grounding: the ability to identify, retrieve, reason over, and visually express explicit or implicit knowledge constraints in a visually verifiable manner. A formal definition and task boundary are provided in Appendix § 8.
Existing work has approached this problem from several directions, including compositional and relation-binding evaluation [16, 9], physical commonsense evaluation [10], factual and knowledge-intensive generation [11, 12], and beyond-prompt or world-knowledge-aware evaluation [13, 14, 15]. While these studies provide important insights, they often focus on specific knowledge types, task formats, or evaluation protocols. There remains a lack of a unified framework that jointly addresses three questions: what constitutes world-knowledge grounding in T2I generation, how to cover both atomic knowledge verification and open-ended knowledge application, and how to evaluate such capabilities with reliable and diagnostic intermediate signals. A more detailed comparison with prior work is provided in Appendix § 9.
To address these issues, we introduce KnowVis, a dual-view diagnostic benchmark for evaluating world-knowledge grounding in T2I models. As summarized in Figure 1, KnowVis combines an L1–L4 hierarchical taxonomy, two complementary benchmark views, and a structured VQA-based MLLM-as-a-Judge protocol. The Skill-Tree Benchmark evaluates controlled L4-level atomic knowledge expression, while the Generalized Benchmark targets open-ended knowledge-grounded generation with human–judge collaborative evaluation.
Figure 1. Overview of the KnowVis framework. It includes iterative taxonomy construction, dual-view benchmark construction, and structured VQA-based evaluation. The Skill-Tree Benchmark focuses on L4-level atomic knowledge diagnosis, while the Generalized Benchmark targets open-ended knowledge-grounded generation with human–judge collaborative evaluation.
Building on this design, our main contributions are as follows:
A unified task formulation. We formalize world-knowledge grounding in T2I generation and define its boundary against general text-image alignment, image quality evaluation, and surface-level instruction following.
A dual-view diagnostic benchmark. We construct KnowVis, consisting of a Skill-Tree Benchmark for atomic knowledge verification and a Generalized Benchmark for open-ended knowledge application, both organized by a shared hierarchical taxonomy.
A structured and attributable evaluation protocol. We propose a VQA-based MLLM-as-a-Judge protocol shared across benchmark views, enabling fine-grained assessment of knowledge correctness, knowledge richness, and diagnostic failure patterns.
Benchmark Framework
Taxonomy and Dual-View Benchmark Design
We define world-knowledge grounding in T2I generation as the ability of a model to identify, retrieve, reason over, and visually express explicit or implicit knowledge constraints required by a prompt. Such constraints may involve factual states, physical laws, symbolic conventions, domain rules, structured relations, or culturally grounded concepts that are not fully specified by surface-level visual descriptions. This differs from general text–image alignment: a prompt such as “a red sphere on a table” mainly tests whether explicitly stated objects, attributes, and spatial relations are rendered, whereas a prompt such as “a convex lens converges parallel light rays” requires the generated image to satisfy an underlying optical rule. We provide the formal task definition and boundary in Appendix § 8.
To organize this capability into a diagnosable evaluation space, we construct an L1-L4 world-knowledge taxonomy. L1/L2 define broad domains and stable subjects, L3 represents scenario-forming knowledge branches, and L4 denotes atomic knowledge points with clear boundaries and stable evaluation criteria. L4 serves as the basis for controlled sample construction, VQA decomposition, and fine-grained attribution, while more subjective or time-sensitive domains are retained at coarser L2/L3 granularity. The taxonomy hierarchy and construction criteria are detailed in Appendix § 8.1 and Appendix § 8.2.
Based on this taxonomy, we construct two complementary benchmark views. The Skill-Tree Benchmark is centered on L4 atomic knowledge points and evaluates whether a model can correctly express specific knowledge constraints under controlled settings. The Generalized Benchmark is centered on L3 knowledge branches and evaluates whether a model can apply relevant knowledge in more realistic, open-ended, and multi-knowledge generation scenarios. Together, the two views support diagnosis from atomic knowledge verification to open-ended knowledge application, while their shared taxonomy enables cross-view analysis and failure attribution.
Benchmark Construction Pipeline
The two benchmark views follow a strategy of separate construction with unified attribution. Skill-Tree cases are constructed around L4 atomic knowledge points through LLM-based candidate generation, automatic filtering, and human screening, so that each retained case has a focused target and clear assessment criteria. Generalized cases are manually curated around L3 branches to preserve realistic, open-ended, and compositional generation scenarios, and are then annotated with potentially involved L4 points for cross-view attribution. This mapping provides partial failure attribution and helps identify taxonomy coverage gaps without assuming that every open-ended failure can be fully explained by existing L4 nodes. Full construction details are provided in Appendix § 10.
Structured Metadata and Benchmark Statistics
To support coverage analysis, capability slicing, and error attribution, we annotate each case with structured metadata. At the knowledge-structure level, Skill-Tree samples are mapped to a single L1–L4 taxonomy path, with L4 as the primary atomic knowledge point. Generalized samples are typically annotated at the L3 branch level, with an additional set of potentially involved L4 points; when stable L3/L4 decomposition is not available, a case may be annotated at the L2 level.
At the task-attribute level, we annotate education level, knowledge operation type, and, for Generalized cases, difficulty. Education level indicates the typical learning stage or knowledge source associated with a case. Knowledge operation type describes the primary way in which knowledge is used in the generation task, including Explicit Knowledge Rendering, Explicit Knowledge Recall, Implicit Knowledge Recall, and Knowledge Reasoning. When multiple operation types are involved, we annotate the primary and highest-level knowledge use. The Skill-Tree Benchmark uses three operation types and does not annotate difficulty, while the Generalized Benchmark uses the full four-way operation schema and additionally annotates difficulty to capture the combined complexity of semantic understanding and visual presentation.The full metadata schema is defined in Appendix § 11.1.
Figure 2. Taxonomy coverage, shown up to the L3 branch level for legibility.
Benchmark statistics and scope. Table 1 summarizes the scale and construction units of the two benchmark views. Skill-Tree contains 1,622 cases covering 240 L4 points, while Generalized contains 550 cases covering 20 L2 subjects and 70 L3 branches. The reported distributions are coverage-driven and evaluability-driven rather than an exhaustive distribution of world knowledge. Full statistics are reported in Appendix § 11.2.
| Statistic | Skill-Tree | Generalized |
| #Cases | 1,622 | 550 |
| Main unit | L4 atomic point | L3 branch |
| Taxonomy coverage | 3 L1 domains<br>240 L4 points | 5 L1 domains<br>20 L2 subjects<br>70 L3 branches |
| Density | 6.8 cases per L4 | – |
| Knowledge operation | 3 types | 4 types |
| Extra metadata | Education level | Education level<br>Difficulty<br>Knowledge quantity |
Table 1. Overview of the two benchmark view.
Evaluation Protocol and Diagnostic Framework
Unified VQA-Based MLLM-as-a-Judge Protocol
Both benchmark views share the same structured judge target: determining whether a generated image satisfies the explicit or implicit world-knowledge constraints required by the prompt. Rather than asking for a holistic judgment of whether an image is “knowledge-correct,” we decompose each case into visually verifiable VQA items. Each item checks a localized piece of evidence, such as the presence of an entity, the correctness of an attribute, the validity of a relation, the binding between annotations and visual elements, or the satisfaction of a knowledge-dependent structure. This decomposition converts open-ended knowledge judgment into structured intermediate signals that can be aggregated and analyzed.
We use three judge types. Required VQAs define the minimal evidence needed for the core knowledge constraint to hold. Optional VQAs define predefined valid supplementary expressions and are used to measure knowledge richness. ADDITIONAL VQAs record valid supplementary expressions that are not covered by the current OPTIONAL set. Since outputs from different systems can contain diverse, open-ended valid knowledge expressions, we dynamically extract ADDITIONAL VQAs during evaluation to capture reasonable supplementary content beyond the predefined OPTIONAL pool; these items are used for analysis and for iteratively refining the OPTIONAL set, rather than being directly included in the main richness score.
Each VQA item is further associated with structured fields for stable judgment and attribution. The component field specifies the visual or annotation element being inspected; knowledge_hint provides background knowledge and judgment criteria; and visual_grounding points the judge to relevant visual evidence or reference cues when needed. Detailed VQA schema definitions and auxiliary annotations for knowledge hints and visual grounding are provided in Appendix § 12. In our implementation, we use Gemini 3.1 Pro Preview as the MLLM judge backend. The judge receives the original prompt, the generated image, and the structured VQA items, and returns a ternary decision for each item, i.e., PASS, FAIL, or NA, together with a brief rationale. Here, NA is used when an optional or additional expression is not applicable or not present in the image, rather than being counted as an incorrect required knowledge constraint.
The protocol design is model-agnostic in the sense that VQA decomposition, schema fields, scoring rules, and aggregation are not tied to a specific MLLM backend. This portability does not imply comparable reliability across judge models with different knowledge, visual perception, and reasoning capabilities. We use Gemini 3.1 Pro Preview as the default judge for the primary benchmark results and assess robustness through a same-setting rerun and alternative strong MLLM judges in § 4.2. A protocol-level comparison with prior automatic evaluation methods is provided in Appendix § 9.
Skill-Tree and Generalized Evaluation
The Skill-Tree Benchmark is designed for scalable diagnosis over L4 atomic knowledge points. Because each case targets a bounded knowledge point with a stable expected visual expression, we use automatic VQA-based evaluation and focus on Required VQAs. A Skill-Tree case is considered correct only when all Required VQAs are satisfied, reflecting whether the generated image correctly expresses the minimal knowledge frame of the target L4 point. For a model and a Skill-Tree case , the case-level correctness score is:
where denotes the Skill-Tree case-level correctness score, is the number of Required VQAs and indicates whether the -th Required VQA is satisfied.
The Generalized Benchmark targets open-ended scenarios where a prompt may involve multiple knowledge constraints, complex contexts, and multiple plausible expression paths. We therefore combine structured judge scores with human MOS ratings instead of using a strict pass/fail criterion alone. Required VQAs are averaged into a knowledge correctness score:
Optional VQAs are used to measure knowledge richness. For each optional item, we set if the corresponding supplementary knowledge expression is present and correct. We set if the expression is present but incorrect, or if it is expected but missing. The knowledge richness score is:
Here, measures the satisfaction of core knowledge requirements, while measures the coverage of predefined supplementary knowledge expressions. For Optional VQAs, both missing expressions and incorrectly expressed knowledge are counted as 0, because they indicate insufficient knowledge richness.
In addition, the Generalized Benchmark evaluates non-knowledge visual instruction following, which measures explicitly stated visual requirements independent of world knowledge, such as subject count, viewpoint, color, layout, and composition. We compute this dimension using an internal visual-instruction judge that decomposes prompts into weighted atomic visual units; details are provided in Appendix § 13.1.1, Eq. (4).
Figure 3 shows the annotation interface used in the Generalized Benchmark. For cross-dimensional analysis, judge-derived scores are linearly mapped to the same numerical scale as human MOS, while image quality, aesthetics, and overall satisfaction are rated directly by human annotators.

Figure 3. Annotation interface.
Knowledge correctness and image quality are evaluated separately. A factual error does not, by itself, lower the image-quality score; a lower score requires an independently observable visual defect, such as blurring, structural distortions, or geometric interpenetration. Overall satisfaction is rated separately as a holistic judgment that may reflect both factual correctness and visual quality, rather than being computed as a fixed weighted combination of the individual dimensions.
Human ratings are collected independently across annotators, followed by discrepancy review when needed. Detailed scoring examples are provided in Table 17, and the conversion of judge-derived scores to the MOS scale is specified in Appendix § 13.1.2. Evaluation rubrics, pre-adjudication inter-annotator agreement, and score aggregation rules are detailed in Appendix § 13.
| Example | Scoring | Reason |
| Case A: Ideal example: Knowledge is correct |
Table 17. Representative Overall Satisfaction Patterns.
Attribution and Diagnostic Analysis
We combine structured evaluation signals with case-level metadata for attribution. The signals include Required-VQA pass/fail results, Optional-VQA coverage, non-knowledge instruction-following scores, and human MOS ratings, while the metadata include taxonomy paths, education level, knowledge operation type, difficulty, and knowledge quantity. This supports failure localization by both evaluation dimension and knowledge-task type. Details are provided in Appendix § 11.1 and Appendix § 14.
For Skill-Tree, we localize failures at the L4 level by aggregating Required-VQA failures across component types and L1–L4 taxonomy levels, with separate tracking of annotation-related failures. For Generalized, we analyze knowledge correctness, richness, instruction following, and human MOS across L2/L3 branches and task metadata to identify open-ended failure patterns.
Experiments
We evaluate KnowVis along three axes: judge reliability and robustness, agreement gains from structured auxiliary fields, and whether the dual-view benchmark reveals world-knowledge grounding failures beyond general T2I capability. We report human–judge agreement, cross-judge and human-correction analyses, component ablations, automatic-metric comparisons, and model performance on the Skill-Tree and Generalized benchmarks.
Experimental Setup
We use three evaluation sets with complementary roles. For Skill-Tree, we sample 480 cases from the full 1,622-case benchmark, covering all 240 L4 atomic knowledge points. We split them into 240 visible development cases for judge-prompt iteration and 240 unseen validation cases for model evaluation and ranking comparison across judges. Human–judge agreement and cross-judge agreement use the full 480-case subset.
For the Generalized Benchmark, we evaluate a 100-case subset of the full 550-case benchmark, selected to cover all L2 subjects and L3 branches. The evaluation follows a human–judge collaborative protocol: the judge assesses knowledge correctness, knowledge richness, and non-knowledge visual instruction following, while human annotators rate image quality, aesthetics, and overall satisfaction. This subset supports analysis of broad model-level trends and representative failure patterns; its limited size does not support precise branch-level rankings or strong conclusions about small differences between models. For General-Domain T2I, we use a 100-case subset from a 600-case pool as a control for general generation capability.
We evaluate overlapping groups of general-purpose, aesthetics-oriented, and text-/knowledge-oriented T2I models.
Each case includes bilingual prompts. For each model, we use the language setting that best matches its documented or observed prompting behavior, while keeping the underlying case, semantic intent, metadata, and evaluation protocol fixed across models. This design reduces language-specific confounds, but we do not interpret the results as a controlled comparison of Chinese-versus-English prompting. Further details are in Appendix § 18.1 and Appendix § 18.
Judge Reliability and Robustness
We first validate Gemini 3.1 Pro Preview, our default judge, against human annotations on the Skill-Tree and Generalized benchmarks. We then examine human–judge agreement and model-ranking stability under repeated evaluation and alternative judge models. Precision and recall throughout the judge-reliability and ablation analyses treat FAIL as the positive class: precision measures the correctness of flagged errors, and recall measures the fraction of human-labeled errors detected.
Default-judge reliability on the Skill-Tree Benchmark. We evaluate the default judge on 480 generated images corresponding to 480 distinct Skill-Tree prompts, with approximately 80 images from each of six T2I models, at two levels: VQA-level agreement measures whether individual checklist judgments match human labels, while case-level agreement measures whether the aggregated pass/fail decision matches the human reference. The default judge achieves 94.51% VQA-level accuracy and 88.12% case-level accuracy, with Cohen’s of 0.8236 and 0.7562, respectively. VQA-level precision and recall are 92.07% and 80.22%. These results support the reliability of the structured judge on the evaluated set, while indicating that some judgment errors remain.
To assess the reliability of the human reference labels used for judge validation, five annotators independently labeled 288 Required VQAs from 50 sampled Skill-Tree cases. Inter-annotator agreement reaches a nominal Krippendorff’s of 0.766, with unanimous agreement on 82.64% of VQAs. These results support the consistency of the human references against which judge performance is assessed. Detailed annotation procedures and agreement distributions are provided in Appendix § 12.4.
Validation in open-ended scenarios. We additionally assess the default judge on 50 sampled cases from the Generalized Benchmark, evaluating Required and Optional VQAs separately. VQA-level accuracy against human annotations reaches 98.41% for Required VQAs and 98.53% for Optional VQAs. These results show high human–judge agreement for both knowledge correctness and knowledge richness on the sampled open-ended cases. Detailed human–judge agreement results for the Generalized Benchmark are reported in Appendix § 16.3.
Contribution of auxiliary fields. We further ablate taxonomy context (category), knowledge hints (knowledge_hint), and visual grounding cues (visual_grounding) while keeping models, samples, VQAs, scoring rules, and aggregation fixed. Removing any auxiliary field reduces accuracy against human annotations. Removing all three fields lowers case-level and VQA-level accuracy by 2.08 and 2.02 percentage points, respectively. The full ablation results are reported in Appendix § 16.2.
Cross-judge validation and ranking robustness. To examine dependence on the default judge, we compare the original evaluation with a same-setting Gemini 3.1 Pro Preview rerun and evaluations using Gemini 3.5 Flash and GPT 5.5. Table 2 summarizes VQA-level agreement with human annotations and ranking consistency across the ten evaluated T2I models.
| Judge setting | Acc. ↑ | Prec. ↑ | Rec. ↑ | ↑ | Kendall’s ↑ | Spearman’s ↑ |
| Gemini 3.1 Pro Preview (original) | 94.51 | 92.07 | 80.22 | 0.8236 | – | – |
| Gemini 3.1 Pro Preview (rerun) | 91.74 | 83.63 | 74.44 | 0.7366 | 0.8222 | 0.9273 |
| Gemini 3.5 Flash | 92.21 | 86.74 | 73.41 | 0.7475 | 0.8222 | 0.9394 |
| GPT 5.5 | 91.33 | 86.03 | 69.11 | 0.7140 | 0.8222 | 0.9394 |
Table 2. Human–judge agreement and ranking consistency across judge settings. Accuracy, precision, and recall are reported as percentages; denotes Cohen’s . Ranking correlations are computed relative to the model ranking obtained with the original Gemini 3.1 Pro Preview evaluation. Higher values indicate stronger agreement or ranking consistency.
Across all tested settings, VQA-level accuracy remains above 91%. Although the rerun and alternative judges yield lower agreement metrics than the original evaluation, they retain substantial agreement with human annotations, with Cohen’s ranging from 0.7140 to 0.7475 and accuracy ranging from 91.33% to 92.21%. These results indicate that agreement with human judgments remains at a comparatively high level despite variation across evaluations.
Model rankings also remain highly consistent with the original ranking: Kendall’s is 0.8222, and Spearman’s ranges from 0.9273 to 0.9394. The top two models remain unchanged across all settings; Gemini 3.5 Flash and GPT 5.5 also preserve the original top-three ordering. No model moves by more than two positions. Together, the maintained human–judge agreement and stable ranking trends support the robustness of the protocol for comparative model evaluation under the tested strong MLLM judge settings. Complete case-level agreement results and per-model rankings are reported in Appendices § 16.4 and § 16.§ 5, respectively.
We further assess the impact of judge errors through a human-correction audit of a total of 50 prompt–image pairs sampled from the Skill-Tree outputs of five T2I models. Correcting the identified errors preserves the top-ranked model and only swaps the second and third positions, yielding Kendall’s of 0.8000 and Spearman’s of 0.9000 between the original and corrected rankings. These results provide additional evidence of ranking stability within the audited subset. Detailed human-correction results are reported in Appendix § 15.3; judge failure modes are analyzed in Appendix § 15.

Figure 4a. L1-domain correlation with human GT VQA pass rate.

Figure 4b. L2-level consistency pattern.
Comparison with Automatic Metrics
We compare our judge-derived VQA pass rate with CLIPScore and VQAScore, using the human-annotated VQA pass rate as the reference. CLIPScore represents embedding-based text–image alignment, while VQAScore represents VQA-based alignment. We report Kendall’s and Spearman’s across L1 domains, and visualize L2-level consistency with human labels. Other preference- or benchmark-specific metrics are discussed qualitatively in Appendix § 9, as their task definitions, inputs, and scoring protocols differ substantially.
As shown in Figure 4, our protocol achieves the strongest correlation with human knowledge labels across all three L1 domains. The gap is most evident in Formal Science, where CLIPScore is negatively correlated with human labels. The L2-level radar plot further shows that structured VQA judgments better preserve human-observed domain-wise patterns, whereas general alignment metrics are less stable across knowledge branches. These results suggest that general text–image alignment metrics are insufficient for world-knowledge correctness evaluation, and motivate our use of knowledge-specific VQA judgments for scalable Skill-Tree evaluation.

Figure 4. Quantitative comparison with existing automatic metrics. (a) Correlation between automatic metrics and human GT VQA pass rate across three L1 knowledge domains. (b) L2-level consistency pattern across 12 subjects, where KnowVis more closely follows the human GT profile than CLIPScore and VQAScore.
Benchmark Results and Diagnostic Analysis
We report model performance on the Skill-Tree and Generalized benchmarks. The goal is not only to compare overall scores, but also to examine whether the benchmark exposes domain- and task-level failure patterns.
Skill-Tree Benchmark. Figure 5 summarizes Skill-Tree results from two complementary views: case-level correctness and VQA-level component pass rates. The case-level results show clear model tiers: Nano Banana 2 performs best, several models fall into a middle range, and Midjourney V7 and Kling Image 3.0 Omni are relatively weaker on Skill-Tree knowledge correctness. The L1 slices further reveal domain-specific variation across Natural Science, Formal Science, and Human Civilization & Society, indicating that world-knowledge grounding cannot be assessed by an overall score alone.

Figure 5. Skill-Tree diagnostic heatmaps. (a) Case-level correctness aggregated by taxonomy and task metadata. (b) VQA-level pass rates across component types and models. Together, the two views show both where knowledge failures concentrate and which structured VQA components are more challenging.
Education level and knowledge operation type provide complementary diagnostic views. Education level should be interpreted as the typical learning stage rather than a direct proxy for generation difficulty, while Knowledge Reasoning more clearly separates weaker models than recall-oriented categories. The VQA-level results further show that entity-related checks are generally easier than relation- and annotation-related checks, suggesting that models often generate relevant objects but fail to express structured knowledge through relations, labels, formulas, arrows, or text–image bindings. A more complete heatmap including L2-level subjects is provided in Figure 8(a); detailed VQA-level model failure analysis is provided in Appendix § 14.§ 2.1.

Figure 8. Full heatmaps for cross-view diagnostic analysis. (a) Skill-Tree case-level correctness across models, taxonomy slices, education levels, and knowledge operation types. (b) Generalized Benchmark overall satisfaction MOS across models and taxonomy slices.
Generalized Benchmark. The Generalized Benchmark evaluates open-ended world-knowledge generation through both judge-derived knowledge dimensions and human MOS dimensions. As shown in Figure 6, Nano Banana 2 leads on multiple dimensions, including knowledge correctness, knowledge richness, instruction following, and overall satisfaction. This suggests that its advantage is not merely driven by perceptual quality, but by stronger performance across both knowledge-centered and general visual dimensions. In contrast, Midjourney V7, despite strong aesthetics, obtains lower knowledge correctness and richness, showing that aesthetic quality does not necessarily imply world-knowledge grounding capability.
Figure 6. Mean Opinion Score (MOS) of different models across evaluation dimensions.
We also observe consistent domain-level patterns between the two benchmark views. As shown in Figure 8(b), models with weak Skill-Tree performance in Formal Science also tend to obtain lower Generalized MOS in corresponding formal-science cases. These observations suggest that L4-level weaknesses can surface in open-ended generation, although the relationship should not be interpreted as a strict causal mapping.
Complementary General-Domain T2I results are reported in Appendix § 16.1. They show that general instruction following, visual quality, and aesthetics are not fully aligned with world-knowledge grounding, supporting the need for dedicated knowledge-grounded evaluation.
Discussion
To examine whether explicit factual guidance improves generation, we conduct a prompt rewriting experiment on 240 unseen validation cases, converting the original prompts into direct visual instructions that supply the relevant factual details. Across the selected cases, the pass rate increases from 0.271 to 0.408, while the Implicit Knowledge Recall subset improves from 0.333 to 0.667 Table 3.
| Slice | Category | Rewritten | Default | |
| Overall | ALL | 0.408 | 0.271 | +0.137 |
| Knowledge operation | Explicit Knowledge Recall | 0.432 | 0.296 | +0.136 |
| Knowledge operation | Implicit Knowledge Recall | 0.667 | 0.333 | +0.334 |
| Knowledge operation | Knowledge Reasoning | 0.263 | 0.132 | +0.131 |
Table 3. Prompt rewriting results across knowledge operation types. Rewriting implicit or underspecified knowledge constraints into direct visual instructions improves the overall case pass rate from 0.271 to 0.408.
These gains suggest that some failures involve knowledge invocation or prompt formulation, rather than rendering alone. However, rewriting also changes instruction-following demands, preventing a clean separation of these factors. Remaining errors may reflect knowledge interpretation, instruction following, or visual rendering. We therefore treat this experiment as a preliminary diagnostic analysis.
Future work will develop controlled prompt pairs targeting the same factual content and evaluation criteria, with relevant facts either unstated or explicitly supplied. Other prompt characteristics and task demands should be matched as closely as possible, and correctness reported separately for each condition. The performance gap would characterize the benefit of factual guidance, but should not be interpreted as a pure measure of knowledge invocation.
Conclusion
We present KnowVis, a dual-view benchmark for diagnosing world-knowledge grounding in T2I models. It combines an L1–L4 hierarchical taxonomy, a Skill-Tree Benchmark for L4-level atomic knowledge diagnosis, and a Generalized Benchmark for open-ended knowledge-grounded generation. Together with a structured VQA-based MLLM-as-a-Judge protocol, KnowVis evaluates whether explicit or implicit knowledge constraints are correctly transformed into visual expressions.
Experiments show that the protocol aligns well with human judgments, better reflects world-knowledge correctness than general automatic metrics, and reveals failures that are not captured by general T2I capability alone. These results suggest that world-knowledge grounding requires dedicated diagnostic evaluation beyond text–image alignment, image quality, or aesthetic preference.
Acknowledgments and Disclosure of Funding
We thank the anonymous reviewers for their constructive feedback. This work was supported by KlingAI Research. The authors declare no additional competing interests.
Task Scope and Taxonomy Definition
Definition of World-Knowledge Grounding in T2I Generation. We define world-knowledge grounding in text-to-image generation as the ability of a model to identify, retrieve, reason over, and visually express the explicit or implicit knowledge constraints required by a prompt. These constraints may involve entities, attributes, relations, processes, rules, factual states, symbolic systems, physical commonsense, or culturally grounded concepts that are not fully specified by surface-level visual descriptions. A generated image is considered world-knowledge-grounded when it satisfies the relevant knowledge constraints and expresses them in a visually verifiable manner.
This notion differs from general text-image alignment, which mainly evaluates whether explicitly mentioned objects, attributes, and relations are rendered in the image. It also differs from generic image quality evaluation, which focuses on realism, aesthetics, perceptual quality, and artifacts rather than the correctness of the underlying knowledge.
Task Boundary. Our benchmark focuses on cases where correct image generation requires more than surface-level prompt rendering, including factual knowledge, structured relations, domain rules, symbolic conventions, physical commonsense, cultural knowledge, or reasoning over implicit constraints. We exclude prompts that primarily test generic image quality, aesthetics, stylistic preference, or low-level visual alignment without requiring world knowledge, such as “a beautiful portrait,” “cinematic lighting,” or “a red cube on a blue table.” Such prompts are included only when they impose additional knowledge constraints beyond the explicitly stated visual elements.
We further distinguish world-knowledge grounding from general instruction following: a model may follow the surface prompt while failing the implicit knowledge requirement, and an image may be visually appealing while still being knowledge-incorrect.
Designing the KnowVis Taxonomy
Hierarchy definition. Table 4 summarizes the functional role of each level in our L1-L4 hierarchical world-knowledge taxonomy.
| Level | Organizational Unit | Definition | Primary Role |
| L1 | Domain | The broadest knowledge category, defining the thematic scope of the taxonomy. Each L1 domain groups a set of related subjects that share a common epistemic nature (e.g., natural sciences, human civilization). | Supports overall coverage analysis and cross-domain capability comparison across the benchmark. |
| L2 | Subject | A stable disciplinary subdivision within an L1 domain. Each L2 subject corresponds to a well-established field of knowledge (e.g., architecture, biology), providing the structural backbone for organizing lower-level branches. | Ensuring a smooth transition from abstract domains to concrete branches. |
| L3 | Branch | A specific knowledge direction within an L2 subject. Concrete enough to define a visual task scenario (e.g., architectural styles, animal anatomy). | Serves as the basis for constructing the Generalized Benchmark. |
| L4 | Knowledge Point | The minimal, atomic unit of knowledge. Each L4 knowledge point has a clearly defined scope, a single verifiable claim, and a stable correctness criterion. | Directly used as the basic unit for Skill-Tree Benchmark construction, MLLM-as-a-judge evaluation, and attribution analysis. |
Table 4. Functional definitions of taxonomy levels L1–L4. The taxonomy progressively constrains and refines agent behavior from L1 to L4, with each level imposing stricter and more granular specifications than the previous.
L4 is the minimal diagnosable knowledge capability unit in the taxonomy. It denotes an atomic knowledge point with clear boundaries, a limited invocation scope, and stable evaluation criteria, so that it can be individually assessed with a clear binary judgment. L4 points serve as the foundation for prompt construction, VQA checklist decomposition, and attribution analysis in the Skill-Tree Benchmark. We therefore exclude open-ended or ambiguous entries when defining L4 points.
Taxonomy Construction Criteria
We construct the world-knowledge taxonomy from encyclopedia-style resources, library knowledge classification systems, commonsense categories, LLM-assisted candidate expansion, and internal human review. L1/L2 levels are organized from broad disciplinary and commonsense categories, while L3/L4 entries are expanded and refined through LLM suggestions and manual review.
To adapt the taxonomy to text-to-image generation, we retain only knowledge points that are visually expressible and judgeable through visual inspection or structured VQA-based criteria. We then iteratively check branch coverage, remove ambiguous or weakly visualizable entries, and fill missing categories.
Related Work
The most relevant prior work falls into three categories: evaluation of image generation models, world-knowledge-oriented text-to-image generation tasks, and structured automatic evaluation.
General-Purpose T2I Evaluation. Early T2I evaluation work primarily centered on two axes: image quality and instruction following. On the preference-comparison side, PickScore [17] and ImageReward [18] learn pairwise human preferences, and HPSv3 [19] further broadens this with a wide-spectrum comparison set and a VLM-based preference model; These methods scale well but produce holistic preference scores with limited interpretability. On the automatic-scoring side, CLIP and CLIPScore [20, 21] measure text–image alignment via embedding similarity with high computational efficiency; GenEval [22] pairs structured prompts with automatic metrics to assess color fidelity and spatial relations; T2I-CompBench and its extended version [16, 23] refine the evaluation granularity for attribute binding, spatial relationships, and object counting; GenAI-Bench [24] introduces large-scale human annotations at the attribute–relation–logic level and demonstrates that VQAScore [8] substantially outperforms CLIPScore on fine-grained alignment. These methods effectively measure image quality, aesthetic performance, and overall instruction following. However, they share a common assumption that all relevant semantic information is explicitly encoded in the prompt. When an evaluation task requires the model to retrieve world knowledge beyond what is described in the prompt, or to perform deep semantic alignment or implicit reasoning, neither preference models nor embedding-similarity methods can reliably distinguish whether generated outputs satisfy factual constraints or commonsense regularities, limiting evaluation confidence.
World-Knowledge-Oriented T2I Benchmarks. As interest in models’ knowledge capabilities has grown, a number of benchmarks targeting specific knowledge types have been proposed. In the direction of factuality and knowledge-intensive evaluation, T2I-FactualBench [11] focuses on models’ ability to memorize and discriminate knowledge concepts; MMMG [25] extends evaluation coverage to ten disciplines—including biology, chemistry, and mathematics—and six educational levels ranging from preschool to PhD, and represents each target image with a knowledge graph of entities and relations; it does not organize knowledge points within each discipline and offers insufficient granularity for assessing specific knowledge points. PhyBench [10] systematically covers physical commonsense across mechanics, optics, thermodynamics, and material properties; T2I-ReasonBench [26] decomposes reasoning ability into four dimensions: idiom interpretation, textual image design, entity reasoning, and scientific reasoning. In the direction of implicit semantics and knowledge beyond the prompt, WISE [13] addresses world knowledge along three axes: cultural common sense, spatio-temporal reasoning, and natural science; ABP [15] explicitly introduces the concept of beyond-prompt knowledge, emphasizing attention to deep semantic consistency; LongT2IBench [27] focuses on semantic relationship generation under long-text conditions. In the direction of broad multi-domain coverage, WorldGenBench [14] organizes human cultural and natural knowledge down to country, region, and natural science subfields; AEGIS [28] further subdivides three macro-areas—STEM, humanities, and everyday life—into 21 sub-domains and introduces multi-dimensional labels to support attribution analysis. Overall, existing benchmarks have individually touched on key aspects such as factuality, common sense, reasoning, and implicit knowledge. However, most focus on a single knowledge type and remain limited in the systematic coverage of knowledge distributions, explicit stratification of difficulty levels, and atomic design of knowledge points. A unified and hierarchically organized model of the "world knowledge" capability space has yet to be established.
Automated Evaluation Methods. At the level of evaluation protocol, automated evaluation has evolved from embedding matching to structured decomposition to large-model judging. Early CLIP-based methods are computationally efficient but have limited discriminative power in complex compositional constraints and multi-step reasoning scenarios. TIFA [7] and DSG [9] substantially improve evaluation granularity and interpretability by decomposing prompts into atomic question-answer units, establishing the foundational paradigm for structured decomposition evaluation. As large model capabilities have advanced, the LLM/MLLM-as-a-Judge paradigm—exemplified by G-Eval, LLaVA-Bench, and MM-Vet [29, 30, 31]—has demonstrated strong potential for evaluating open-ended outputs. Further, WorldGenBench [14] and AEGIS [28] incorporate checklist mechanisms to improve evaluation stability; GenEval 2 [32] mitigates the tendency of purely atomic evaluation to diverge from holistic human judgment by combining atomic and open-ended evaluation via Soft-TIFA; GIR-Bench [33] reveals the gap between cross-task knowledge transfer and faithful generation through understanding–generation consistency with explicit logical constraints; UniGenBench++ [34] provides a unified semantic assessment over hierarchical criteria, with point-wise scores that make individual test points interpretable. These advances demonstrate the practical feasibility of automated evaluation for T2I tasks. Nevertheless, world-knowledge-grounded generation imposes higher demands on automated evaluation than general instruction-following tasks: assessing correctness requires accurate verification against factual constraints, commonsense regularities, or implicit background knowledge, and the evaluation criteria themselves must be grounded in verifiable knowledge sources. Without structured evaluation design and reliability validation mechanisms, automated judgment results are difficult to trust and cannot support fine-grained attribution analysis.
Table Table 5 provides a systematic comparison of representative T2I world-knowledge benchmarks. As shown, existing works exhibit consistent limitations in both knowledge coverage systematicity and automatic evaluation reliability.
| Benchmarks | Taxonomy * | Dataset Structure | Evaluation Protocol † | Attribution ‡ |
| N F H A R | Atomic Generalized | Form. MLLM. Human Val. | Diff. & Edu. Text/Anno. | |
| WISE [13] | ✓ ✗ ✓ ✗ ✗ | ✗ ✓ | Score-based ✓ ✓ | ✗ ✗ |
| ABP [15] | ✓ ✗ ✓ ✓ ✗ | ✗ ✓ | Score-based ✓ ✓ | ✗ ✗ |
| T2I-ReasonBench [26] | ✓ ✓ ✗ ✓ ✗ | ✗ ✓ | Score-based ✗ ✓ | ✗ ✗ |
| GIR-Bench (T2I) [33] | ✓ ✓ ✓ ✗ ✗ | ✗ ✓ | Score-based ✓ ✓ | ✗ ✗ |
| UniGenBench++ [34] | ✗ ✓ ✓ ✓ ✗ | ✗ ✓ | Score-based ✓ ✗ | ✗ ✗ |
| T2I-FactualBench [11] | ✓ ✓ ✗ ✗ ✗ | ✓ ✗ | Score-based ✓ ✓ | ✗ ✗ |
| PhyBench [10] | ✓ ✗ ✓ ✗ ✗ | ✓ ✗ | Score-based ✓ ✓ | ✗ ✗ |
| MMMG [25] | ✓ ✓ ✓ ✗ ✗ | ✓ ✗ | Graph-based ✓ ✓ | ✗ ✗ |
| WorldGenBench [14] | ✓ ✓ ✓ ✗ ✗ | ✗ ✓ | Checklist-based ✓ ✗ | ✓ ✗ |
| AEGIS (T2I) [28] | ✓ ✓ ✓ ✓ ✗ | ✗ ✓ | Checklist-based ✓ ✓ | ✗ ✗ |
| KnowVis (Ours) | ✓ ✓ ✓ ✓ ✓ | ✓ ✓ | Checklist-based ✓ ✓ | ✓ ✓ |
Table 5. Systematic comparison of existing T2I world-knowledge benchmarks across coverage distribution, benchmark hierarchy, evaluation protocol, and attribution labeling, revealing shared limitations in knowledge coverage systematicity and automatic evaluation reliability.
In summary, existing work has made substantive progress along two separate directions—breadth of knowledge coverage and automated evaluation protocols—yet a systematic integration of the two remains absent. Knowledge-oriented benchmarks largely lack unified modeling of the knowledge capability space and hierarchically structured benchmark design, while existing automated evaluation protocols have yet to simultaneously achieve reliability, interpretability, and fine-grained attribution in world-knowledge tasks. Our work sits at the intersection of these two lines of research, aiming to construct a unified world-knowledge benchmark framework paired with an attributable, structured automated evaluation protocol—one that jointly supports atomic capability diagnosis, hierarchical knowledge organization, and generalization analysis in open, complex scenarios.
Benchmark Construction Details
We construct two independent yet logically aligned data pipelines to support the Skill-Tree Benchmark and the Generalized Benchmark, respectively. Both benchmarks share the same L1–L4 hierarchical taxonomy, but their evaluation objectives, sample organization, knowledge granularity, and intended use cases differ fundamentally, making a unified construction pipeline inappropriate.
The Skill-Tree Benchmark is designed for systematic diagnosis of model capabilities. Its central questions are: Does the model possess a given type of atomic world knowledge? At which knowledge points does it fail consistently? How do different models compare on the same knowledge point? Accordingly, this benchmark requires broad coverage of L4 atomic knowledge points, with each point anchored by samples that are decidable, attributable, and amenable to large-scale statistical analysis. Its construction therefore follows a bottom-up, L4-focused batch generation approach: LLM-driven prompt generation centered on atomic knowledge points, followed by automated filtering and human review.
The Generalized Benchmark is designed to assess model performance on realistic, complex tasks. Its central question is: When a user issues a complex, natural, open-ended generation request, can the model draw on multiple knowledge points simultaneously and produce correct, rich, and coherent knowledge expressions in the generated image? Real user prompts rarely isolate a single knowledge point; they typically involve entities, attributes, relations, cultural context, physical constraints, and contextual reasoning in combination. As such, the Generalized Benchmark is not suited to direct bottom-up assembly from L4 points. Instead, its samples are crafted manually around L3 knowledge branches, with L4 annotations added after sample completion, ensuring that even complex tasks remain traceable within the unified knowledge attribution framework.
We therefore adopt a strategy of separate construction with unified attribution: both data streams originate from the same taxonomy but follow distinct construction pipelines according to their evaluation objectives, and both ultimately converge on a shared evaluation and attribution framework via L4 labels and VQA checklists. This design ensures that the Skill-Tree Benchmark achieves high coverage, scale, and diagnostic precision, while the Generalized Benchmark preserves the complexity, compositional diversity, and ecological validity required for realistic evaluation scenarios.
Skill-Tree Benchmark Synthesis
The Skill-Tree Benchmark is constructed via a pipeline of LLM-based automatic generation, LLM-based review and filtering, and human screening and rewriting. The overall workflow proceeds as follows:
An LLM generates candidate prompts in batch according to predefined knowledge points.
A separate model performs automated review and filtering on the candidates:
Samples that pass review enter the result pool.
Samples that fail, along with their identified failure reasons, are fed back into the next generation iteration.
Generation and filtering form a closed-loop feedback process, which is iteratively repeated until the result pool reaches the target size.
Automated Filtering Criteria. The automated filtering stage is designed to remove candidate prompts that are difficult to evaluate reliably or insufficiently aligned with the target L4 knowledge point. A prompt is accepted only if it clearly targets the current L4 knowledge point, can be visually executed, and does not violate predefined format or content constraints. Conversely, a prompt is filtered or marked for revision if it is off-topic, visually unverifiable, overly complex, conflates multiple knowledge points, or violates the required prompt format. We describe the main filtering criteria below and provide representative examples of candidates rejected by the reviewer model.
Knowledge alignment: must be anchored to the current L4. The core subject of a prompt must correspond to the L4 leaf node in the current taxonomy. It should not drift toward other knowledge points, other disciplinary concepts, or content on the exclusion list. If the prompt introduces knowledge unrelated to the current L4, or touches content prohibited under “prompt types to exclude,” it should be filtered or marked for revision.
Example: filtered by the reviewer model
Target taxonomy: Natural Science → Physics → Mechanics → Concurrent Force Equilibrium.
Candidate prompt: Draw a force diagram of a wooden block resting on an inclined plane held by a string, and label the equilibrium of concurrent forces.
Filtering reason: Although the prompt is related to concurrent force equilibrium, it relies on an inclined-plane force diagram and may introduce multiple confounding mechanics concepts, including gravity, friction, support force, tension, and force decomposition. This makes the evaluation target less focused and increases visual complexity and attribution ambiguity for a single L4 knowledge point.
Revision suggestion: Rewrite the prompt into a more focused static-equilibrium scene, such as an educational illustration of a decorative lamp suspended by three strings connected at the same point and remaining stationary, with a label indicating concurrent force equilibrium.
Visual executability: must be drawable. A prompt must be translatable into a concrete image, scene, experiment, diagram, or visualization. Prompts that are purely theoretical, purely abstract, purely derivational, philosophical, or lack a visual subject should be filtered or marked for revision.
Example: filtered by the reviewer model
Target taxonomy: Human Civilization and Society → Humanities and Architecture → Architectural Styles → Renaissance Architecture.
Candidate prompt: Show the front elevation of the Villa Rotonda, using geometric auxiliary lines to outline its symmetry.
Filtering reason: The prompt asks the model to visualize an abstract design principle rather than a concrete architectural component. Although symmetry is relevant to Renaissance architecture, “outlining symmetry” is an open-ended analytical requirement and lacks a clearly decidable visual target.
Revision suggestion: Shift the focus from an abstract property to a concrete physical component, such as: draw the front elevation of the Villa Rotonda and label the triangular pediment at the top with its name.
Atomicity: only one primary knowledge point should be tested. A prompt should involve a single, well-defined knowledge call. It should not test multiple coordinate concepts within the same image, require a complete depiction of a complex system, or rely on extensive specialist annotations. Prompts that simultaneously test multiple knowledge points or make failure attribution difficult should be filtered or marked for revision.
Example: filtered by the reviewer model
Target taxonomy: Natural Science → Biology → Organs and Basic Functions → Animal Organs and Their Basic Functions.
Candidate prompt: Draw an anatomical diagram of a fish and label the name of the core organ responsible for gas exchange.
Filtering reason: The prompt requires a fish anatomical diagram, which may involve multiple organs and internal structures beyond the target knowledge point. This makes the evaluation target less atomic and increases the difficulty of attributing errors to a single organ-function relation.
Revision suggestion: Simplify the visual carrier to a single organ or local region, such as: draw a structural diagram of fish gills and label their function in gas exchange, or draw a complete fish and circle the location of the gills.
Clarity and universality: specific expression, universally recognizable carrier. A prompt must clearly specify what object, scene, or diagram should be generated. It should avoid vague expressions such as “some,” “various,” “roughly,” or “depict a certain situation.” The visual carrier should rely on broadly recognizable objects or diagrams, rather than niche cultural items or highly domain-specific carriers that may introduce knowledge conflation.
Example: filtered by the reviewer model
Target taxonomy: Natural Science → Physics → Thermodynamics → Calorific Value and Heat Engine.
Candidate prompt: Draw a Sankey diagram of heat engine energy flow, accurately labeling useful work and dissipated heat.
Filtering reason: The prompt requires an accurate Sankey diagram but does not specify the underlying energy distribution. Without concrete values for useful work, exhaust losses, and dissipated heat, the visual solution is underdetermined and difficult to evaluate consistently.
Revision suggestion: Provide specific energy distribution values, such as total heat input 100%, useful work 30%, exhaust heat loss 40%, and thermal dissipation 30%, and specify where the labels should appear.
Structural regularity: short, single-sentence, no answer spoilers. Prompts should be concise, structurally simple, and within the preset length limit. Compound prompt structures using conjunctions such as “and,” “also,” or “moreover” should be avoided when they introduce additional requirements. A prompt should also avoid directly revealing the target answer or final knowledge outcome, unless the task is explicitly designed as knowledge visualization rather than knowledge recall.
Example: filtered by the reviewer model
Target taxonomy: Formal Science → Mathematics → Numbers and Calculation → Derivatives and Differentials.
Candidate prompt: Draw a freely falling ball, and write the formula beside it.
Filtering reason: The prompt combines multiple requirements in one sentence and directly provides the key formula in the prompt. This weakens the intended knowledge-recall or knowledge-grounding objective and makes the evaluation less diagnostic.
Revision suggestion: Rewrite the prompt into a more focused task that does not directly reveal the target answer, or explicitly treat it as a knowledge-visualization task if the formula is intended to be provided.
Human Screening and Rewriting Criteria. Human screening and rewriting is conducted as a subsequent refinement stage after automated filtering. While the automated reviewer model removes candidates with obvious violations, such as off-topic content, non-visualizability, excessive complexity, or format errors, human screening further focuses on borderline cases that require semantic judgment, evaluation stability, and benchmark-level consistency. In other words, automated filtering serves as a first-pass quality gate, whereas human screening and rewriting acts as a second-pass refinement step to improve evaluability, attributability, and stability of the Skill-Tree Benchmark.
The human screening and rewriting process follows the principles below.
Single-variable principle. Each L4 knowledge point should focus on exactly one key variable, avoiding the inclusion of multiple core examination targets within the same sample. This reduces judgment interference and improves capability isolation.
Visualization constraint. Each selected knowledge point must be clearly mappable to image content — i.e., it must have a well-defined visual expression basis. Since the core objective of the Skill-Tree Benchmark is to assess the model’s knowledge-to-image mapping ability, knowledge content that is difficult to visualize or that primarily requires textual reasoning for evaluation is not suitable for inclusion.
Decidability principle. Each sample must have a clear and stable ground truth, capable of reliably distinguishing correct knowledge expression from incorrect knowledge expression, while avoiding cases where evaluation boundaries are ambiguous or subjective interpretation dominates.
Disambiguation principle. Sample construction should minimize ambiguous expressions and multiple interpretable readings, ensuring that the knowledge boundary targeted by the prompt is clear and semantically unambiguous, thereby reducing evaluation noise introduced by linguistic ambiguity.
High-coverage principle. Knowledge point selection should prioritize the following:
High-frequency knowledge: knowledge content that appears more commonly in real-world use scenarios and is more salient to users;
General commonsense: foundational knowledge with broad applicability;
Cross-culturally stable knowledge: knowledge whose content and evaluation criteria remain relatively consistent across different cultural contexts.
Examples of Human Rejection after Automated Acceptance. The following examples illustrate candidates that passed automated filtering but were removed during human screening. These cases are not obviously invalid at the surface level: they are generally aligned with the target L4 knowledge point and have a drawable visual carrier. However, human screening further removes them because they violate higher-level requirements such as variable isolation, decidability, or semantic disambiguation.
Violation of the single-variable principle. A sample should focus on a single core variable associated with the target L4 knowledge point. If a prompt mixes the target knowledge with unrelated generation abilities such as slide design, layout design, formula rendering, or excessive visual composition, it may reduce attribution clarity and should be removed or rewritten.
Example: accepted by automated filtering but rejected by human screening
Target taxonomy: Natural Science → Physics → Mechanics → Pressure.
Candidate prompt: Design a physics course slide with the formula centered on the page and draw a schematic diagram of water depth.
Reason for automated acceptance: The prompt is clearly related to the target knowledge point of pressure. The formula and the schematic diagram of water depth both directly correspond to the relationship between liquid pressure and depth. The visual target is explicit, and the prompt does not obviously drift toward unrelated mechanics concepts.
Reason for human rejection: Although the prompt is semantically aligned with pressure, the actual generation target mixes multiple abilities, including formula rendering, course-slide layout, diagram composition, and physical relation visualization. For an atomic Skill-Tree sample, a failure may come from text rendering or slide formatting rather than from misunderstanding the pressure knowledge point itself. Therefore, the sample is removed under the single-variable principle.
Violation of the decidability principle. A sample should have a clear and stable expected outcome. If multiple visual outputs can be considered reasonable under the same prompt, or if the correctness boundary cannot be reliably determined, the sample should be removed or rewritten.
Example: accepted by automated filtering but rejected by human screening
Target taxonomy: Natural Science → Physics → Mechanics → Buoyancy.
Candidate prompt: Show the most reasonable posture of a person moving freely in the Dead Sea.
Reason for automated acceptance: The prompt is related to the knowledge that the high salinity of the Dead Sea makes human bodies easier to float. It has a clear visual subject, namely a person in the Dead Sea, and can be converted into an image-generation task. It also does not directly reveal the expected result of floating, which makes it superficially suitable for testing knowledge-driven generation.
Reason for human rejection: The expression “moving freely” can correspond to many visually plausible states, such as floating, standing, swimming, walking, sitting, or playing in water. The phrase “most reasonable posture” also lacks a stable and unique evaluation criterion. As a result, it is difficult to reliably determine whether the generated image correctly expresses the target buoyancy-related knowledge. The sample is therefore removed because it does not satisfy the decidability principle.
Violation of the disambiguation principle. A sample should avoid ambiguous wording and underspecified conditions. If the prompt admits multiple interpretations that correspond to
different expected answers, the sample should be removed or rewritten to make the intended knowledge target explicit.
| Example: accepted by automated filtering but rejected by human screening |
| Target taxonomy: Natural Science Climate and Astronomy Day-Night and Time Change Time Zone and Local Time Calculation. |
| Candidate prompt: Show clock faces for Tokyo and Los Angeles in the morning. |
| Reason for automated acceptance: The prompt uses clock faces as a clear visual carrier and appears to test the representation of local time or time-zone difference between Tokyo and Los Angeles. The knowledge target is relatively focused, and the visual output can in principle be judged through the displayed clock times and their city associations. |
| Reason for human rejection: The prompt does not specify whether “morning” refers to morning in Tokyo, morning in Los Angeles, or the morning time in both cities separately. Without a reference time such as “when it is 8:00 a.m. in Tokyo,” the task may be interpreted as drawing two independent morning clocks rather than performing a time-zone conversion. This ambiguity makes the expected answer underdetermined, so the sample is removed under the disambiguation principle. |
Table Example.
Generalized Benchmark Curation
Unlike the Skill-Tree Benchmark, the Generalized Benchmark targets more open-ended and realistic generation scenarios. Therefore, its cases are not constructed around a single L4 atomic knowledge point as the only center. Instead, they are manually designed and curated primarily around L3 knowledge branches. Each generalized case is typically associated with a target L3 branch. After the case is finalized, we further annotate the set of potentially involved L4 atomic knowledge points, which supports subsequent cross-view analysis and error attribution.
During manual construction, we follow the principles below.
Alignment with the Target L3 Branch. Each generalized case should first be clearly aligned with a target L3 branch. We use L3 as the main organizational unit for open-ended scenario construction because it is more suitable than L4 for supporting complex prompts, while being more specific than L1/L2 and thus maintaining clear task boundaries.
For example, under Natural Science Physics Mechanics, we can construct open-ended scenarios involving mechanical phenomena such as force, motion, equilibrium, pressure, and buoyancy. Under Formal Science Mathematics Geometric Figures, we can construct open-ended scenarios involving geometric objects, angular relations, plane figures, solid geometry, volume, or geometric formulas. Such cases are not required to correspond to only one L4 atomic knowledge point. Instead, they may naturally involve multiple related L4 points within the same L3 branch.
Branch Representativeness. Cases should cover knowledge scenarios that are representative, common, and diagnostically valuable within the target L3 branch, rather than overly marginal or artificially designed cases.
For example, within Physics Mechanics, representative cases may jointly examine L4 atomic knowledge points such as gravity, friction, pressure, Newton’s laws, torque equilibrium, and force composition and decomposition. These L4 points are common and visually expressible within Mechanics, making them suitable as potential knowledge anchors in generalized cases.
Task Diversity. The Generalized Benchmark should not be limited to a single prompt format, such as formula diagrams or instructional slides. During manual construction, we cover diverse real-world generation needs, including instructional diagrams, science posters, realistic scene illustrations, structural diagrams, annotated explanatory figures, comparison diagrams, and integrated explanatory visuals.
For example, cases under Physics Mechanics may include both instructional diagrams for mechanics and real-world scenes illustrating equilibrium, buoyancy, or pressure. Cases under Mathematics Geometric Figures may include geometric structure diagrams as well as learning-oriented illustrations with formulas, angles, side lengths, or volume annotations. This diversity prevents the benchmark from evaluating only one type of visual capability.
and enables a broader assessment of world-knowledge grounding in open-ended generation tasks.
4. Supplementary L4 Annotation. Although the Generalized Benchmark is primarily constructed around L3 branches, we supplement each finalized case with annotations of the potentially involved L4 atomic knowledge points. This supplementary L4 annotation is not intended to forcibly decompose a generalized case into single-point tests. Instead, it establishes a mapping between the Generalized Benchmark and the Skill-Tree Benchmark, allowing us to analyze whether failures in open-ended scenarios are associated with weaknesses in specific foundational L4 atomic knowledge points.
5. Cases Annotated Only at the L2 Level. For some domains, the boundaries of L3/L4 may be unstable, or the knowledge may be difficult to decompose into objective and visually judgeable atomic points. In such cases, the Generalized Benchmark allows a small number of samples to be annotated only at the L2 level.
Typical examples include Arts and Media Visual Arts and Arts and Media Design and Communication. Knowledge in these domains often involves style, aesthetics, cultural context, and open-ended expression. Compared with domains such as physics and mathematics, they may not be stably decomposable into clearly bounded L4 atomic knowledge points that can be judged through a small set of VQAs. Forcibly decomposing them may introduce substantial subjectivity.
Examples of Generalized Benchmark Curation. The following examples illustrate how generalized cases are manually curated around target L3 branches and then annotated with potentially involved L4 knowledge points. Unlike the Skill-Tree Benchmark, which focuses on a single L4 atomic knowledge point, the Generalized Benchmark allows a prompt to involve multiple related L4 points under the same L3 branch, as long as the task remains representative, visually assessable, and close to realistic open-ended generation scenarios.
Example 1: Mechanics-oriented generalized case
Target L3 branch: Natural Science Physics Mechanics.
Prompt: Draw an adult and a child on both sides of a wooden door pushing it in opposite directions to keep it stationary, with the adult close to the door hinge and the child far away, using arrows to mark the direction of their pushing forces.
Potential L4 annotations: Torque Equilibrium; Force Composition and Decomposition; Concurrent Force Equilibrium.
Curation rationale: This case is aligned with the Mechanics branch and represents a realistic physical scenario rather than an isolated textbook formula. The setting of an adult pushing near the hinge and a child pushing farther away implicitly tests whether the model can express the effect of force arm length on rotational equilibrium. The requirement to mark pushing directions with arrows further introduces structured visual annotation. Therefore, this sample is suitable for the Generalized Benchmark because it combines scenario realism, L3 alignment, multiple related L4 knowledge points, and moderate visual reasoning complexity.
Cross-view Iterative Refinement.
The two views are complementary through a cross-view iterative refinement loop. The Skill-Tree Benchmark provides an attributable and localized profile of atomic knowledge capabilities, which helps explain the sources of failure in complex open-ended cases from the Generalized Benchmark. Conversely, the Generalized Benchmark exposes new knowledge combinations, implicit constraints, and expression patterns that emerge in realistic open-ended scenarios. These observations help identify coverage gaps in the Skill-Tree taxonomy and further drive the supplementation and refinement of L3/L4 categories.
Annotation Schema and Benchmark Statistics
Case-level Metadata Schema
To support cross-view analysis, capability slicing, and error attribution, we define a unified case-level metadata schema for both the Skill-Tree Benchmark and the Generalized Benchmark. The schema describes each case along three task attributes: knowledge operation type, difficulty, and education level.
Skill-Tree only uses three categories under knowledge operation type and does not use difficulty. Generalized uses four categories under knowledge operation type and additionally annotates difficulty. Education level is used in both benchmark views. This shared schema keeps the two views aligned under a common annotation language, while allowing each view to preserve its own evaluation focus.
Knowledge Operation Type. Knowledge operation type describes the main way in which a model is expected to use world knowledge when completing a case. If multiple knowledge operation types are involved, we annotate the primary and highest-level knowledge use.
Explicit Knowledge Rendering. The prompt directly provides the expected answer or knowledge content, and the model is mainly required to visualize it. This includes directly expressing scientific principles, entity names, quantitative relations, or logical processes through non-natural indicative elements such as text, numbers, symbols, arrows, scales, or callouts. It also includes cases where the prompt fully specifies the inherent properties of a knowledge-intensive entity and only requires visual rendering.
Explicit Knowledge Recall. The prompt directly specifies the name of a knowledge-intensive entity, concept, or phenomenon. The model can generate the expected visual form by recalling the corresponding keyword and its associated visual patterns from training data, without necessarily understanding the underlying mechanism, related knowledge, or real-world simulation process.
Implicit Knowledge Recall. The prompt does not directly mention the target knowledge point. Instead, it describes scene context, initial states, actions, typical appearances, or interaction conditions, requiring the model to indirectly recall the relevant physical state, interaction outcome, scientific phenomenon, or knowledge concept from world knowledge and visualize it.
Knowledge Reasoning. Beyond explicit or implicit recall, the model needs to reason, decompose, judge, or calculate in order to apply knowledge to a specific problem. When a case involves both recall and reasoning, we annotate it as Knowledge Reasoning because it represents a higher-level form of knowledge use.
Difficulty. Difficulty describes the overall answering difficulty of a generalized case and is used only in the Generalized Benchmark. It is annotated by jointly considering semantic complexity and presentation complexity. Semantic complexity captures noise, long-range dependencies, implicit conditions, abstract descriptions, or metaphors in the prompt. Presentation complexity captures how difficult it is to express the knowledge in visual form, such as transforming a simple scene into a complex cutaway diagram, arranging multiple entities, binding annotations to visual elements, or organizing formulas, arrows, and explanatory structures.
We use three levels: high, medium, and low. High-difficulty cases usually involve complex semantics and complex visual presentation. Medium-difficulty cases have relatively clear knowledge targets but still require non-trivial scene understanding, knowledge composition, or visual organization. Low-difficulty cases have stable knowledge targets and simple visual forms.
Difficulty is not the same as education level. Difficulty describes the complexity of a specific generation task, while education level describes the typical learning stage or knowledge source of the involved knowledge.
Education Level. Education level indicates the typical learning stage or knowledge source associated with a case. It is used to analyze model performance across different levels of knowledge, rather than to directly represent task difficulty.
We use five labels: primary school, middle school, high school, university, and professional domain. The first four correspond to knowledge typically learned at the respective educational stages. Professional domain refers to knowledge that is not part of general education, but is usually acquired through professional training, occupational practice, industry experience, or social contexts. Examples include national standards, industry procedures, professional roles, specialized skills, social conventions, traditional customs, public-space signage, and architectural functional zones.
Benchmark Statistics
Based on the case-level metadata schema defined above, we summarize the benchmark statistics from two perspectives: taxonomy coverage and task-attribute distribution. These statistics are intended to verify the coverage of the two benchmark views, their cross-view alignment, and the diagnostic utility of the proposed annotation schema.
L1 Domain Distribution. Table 6 reports the L1 domain distribution of both views. The two views share a similar major-domain trend: Natural Science, Human Civilization and Society, and Formal Science are the primary domains in both benchmarks. This indicates that the two views are aligned under the same world-knowledge taxonomy. At the same time, the Generalized Benchmark further includes Arts and Media and Real-time Dynamic Events, which are more open-ended and less suitable for stable L4-level atomic decomposition.
| L1 Domain | Skill-Tree | Skill-Tree | Generalized | Generalized |
| Count | Percent | Count | Percent | |
| Natural Science | 841 | 51.8% | 224 | 40.7% |
| Human Civilization and Society | 489 | 30.1% | 180 | 32.7% |
| Formal Science | 292 | 18.0% | 71 | 12.9% |
| Arts and Media | - | - | 50 | 9.1% |
| Real-time Dynamic Events | - | - | 25 | 4.5% |
Table 6. L1 domain distribution of the Skill-Tree and Generalized benchmarks.
Skill-Tree Task Attributes. Table 7 reports the task-attribute distribution of the Skill-Tree Benchmark. Since Skill-Tree is centered on L4 atomic knowledge points, it does not annotate difficulty. It uses three knowledge operation types: Explicit Knowledge Recall, Implicit Knowledge Recall, and Knowledge Reasoning. The distribution is dominated by Explicit Knowledge Recall, which is consistent with the goal of testing whether models can reliably recall and visualize canonical L4 knowledge concepts. The benchmark also includes Knowledge Reasoning and a small number of Implicit Knowledge Recall cases, enabling analysis of whether models can infer or apply atomic knowledge beyond direct concept-name recall.
| Attribute | Sub-label | Count | Percent |
| Education level | Primary school | 166 | 10.2% |
| Education level | Middle school | 715 | 44.1% |
| Education level | High school | 435 | 26.8% |
| Education level | University | 99 | 6.1% |
| Education level | Professional domain | 207 | 12.8% |
| Knowledge operation type | Explicit Knowledge Recall | 1,322 | 81.5% |
| Knowledge operation type | Implicit Knowledge Recall | 33 | 2.0% |
| Knowledge operation type | Knowledge Reasoning | 267 | 16.5% |
Table 7. Task-attribute distribution of the Skill-Tree Benchmark.
Generalized Task Attributes. Table 8 reports the task-attribute distribution of the Generalized Benchmark. Compared with Skill-Tree, Generalized uses the full four-way knowledge operation schema and additionally annotates difficulty and knowledge quantity. The benchmark contains 126 Knowledge Reasoning cases and 235 compositional-knowledge cases, showing that it evaluates not only direct knowledge recall but also higher-level knowledge use and multi-knowledge composition. In addition, professional-domain cases account for a large proportion, reflecting that open-ended T2I generation often involves knowledge from professional, social, product, and public-space contexts rather than only school-based knowledge.
| Attribute | Sub-label | Count | Percent |
| Difficulty | Low | 293 | 53.3% |
| Medium | 176 | 32.0% | |
| High | 81 | 14.7% | |
| Education level | Primary school | 27 | 4.9% |
| Middle school | 170 | 30.9% | |
| High school | 94 | 17.1% | |
| University | 10 | 1.8% | |
| Professional domain | 249 | 45.3% | |
| Knowledge operation type | Explicit Knowledge Rendering | 30 | 5.5% |
| Explicit Knowledge Recall | 372 | 67.6% | |
| Implicit Knowledge Recall | 22 | 4.0% | |
| Knowledge Reasoning | 126 | 22.9% | |
| Knowledge quantity | Single | 315 | 57.3% |
| Compositional | 235 | 42.7% |
Table 8. Task-attribute distribution of the Generalized Benchmark.
| Education Level | Low | Medium | High | Total |
| Primary school | 18 | 9 | 0 | 27 |
| Middle school | 120 | 40 | 10 | 170 |
| High school | 52 | 24 | 18 | 94 |
| University | 5 | 1 | 4 | 10 |
| Professional domain | 98 | 102 | 49 | 249 |
| Total | 293 | 176 | 81 | 550 |
Table 9. Education level difficulty in the Generalized Benchmark.
Cross-tabulation Analysis. The cross-tabulation 9 between education level and difficulty shows that difficulty is not equivalent to education level. Professional-domain cases span low, medium, and high difficulty levels, and school-level knowledge can also appear in medium or high-difficulty generation tasks. This supports the design choice in our metadata schema: education level describes the typical learning stage or knowledge source, whereas difficulty captures the overall complexity of a specific generation task.
Current Limitations and Future Refinement. The current benchmark statistics also reveal several directions for refinement. First, the domain distribution should not be interpreted as an unbiased distribution of world knowledge. Our sampling is coverage-driven and evaluability-driven: domains such as Natural Science and Human Civilization and Society are emphasized because they more often support objective visual judgment and structured attribution. Future versions can further balance domain coverage while preserving evaluability.
Second, Real-time Dynamic Events currently contain a relatively small number of cases. This is because time-sensitive knowledge requires explicit timestamping, source verification, and update mechanisms, which are more difficult to stabilize than static knowledge. Future extensions should increase this subset with clearer temporal references and reproducible evaluation criteria.
Third, the Generalized Benchmark contains relatively fewer primary-school and university-level cases. Primary-school knowledge is often visually simple and may overlap with common everyday scenes, making it less diagnostic in open-ended generation unless the prompt introduces richer visual or contextual constraints. University-level knowledge, by contrast, is often abstract, specialized, or difficult to express visually in a stable manner. Future versions can expand both categories by constructing more representative open-ended scenarios while maintaining visual judgeability and clear evaluation criteria.
Structured Judge Protocol and Automatic Evaluation
VQA Schema Definition
In the benchmark defined in this paper, each prompt is decomposed into a variable number of VQA items. The field definitions are as follows:
Question: The Question is the core element of each VQA item, describing the visual or semantic aspect being assessed. Questions are always formulated as affirmative declarative statements. When the generated image conforms to the statement, the item is judged as PASS; otherwise, it is judged as FAIL. For questions of type OPTIONAL, when the generated image does not contain the potentially expected content at all, the item is judged as NA.
Judge Type: Judge Type specifies the necessity of verification for a given VQA item, and directly governs how VQA-level judgments are aggregated into case-level MOS scores for knowledge correctness and knowledge richness. Table 10 shows detailed definition of Judge Type
| Judge Type | Definition | Affected Metrics |
| REQUIRED | Mandatory items. Elements explicitly required by the original prompt, or conditions that are physically absolute and implicitly necessary to realize the described scene or knowledge point in the physical world. Example: a prompt requesting "a submarine underwater" must include an underwater environment; a prompt requesting "an apple falling" must depict the apple airborne with a downward trajectory. | Knowledge Correctness |
| OPTIONAL | Optional items. Elements not explicitly required by the original prompt, but highly likely to be spontaneously generated by T2I models when processing knowledge-intensive tasks—most typically: additional textual annotations, arrows, auxiliary diagrams, or explanatory labels. The judgment logic for this type must follow a "penalize only if attempted incorrectly" principle: items are silently marked NA if absent, and marked FAIL only if generated incorrectly. Common failure cases: a model asked to plot a function curve that adds axis-point annotations with incorrect coordinates; a model depicting a historical scene whose surrounding environment is anachronistic. | Knowledge Richness |
| ADDITIONAL | Supplementary items. Like OPTIONAL, but derived from multimodal understanding of the actually generated image: additional elements spontaneously produced by the T2I model are identified, converted into verification checkpoints that define their correct appearance, and appended as supplementary VQA items. | None |
Table 10. Detailed definition of VQA Judge Type. Shows how different Judge Type affect metrics.
Component: Component specifies the linguistic constituent of the prompt that the VQA item corresponds to. We organize components into two kinds and seven component types. VQA items are first divided by whether they assess entities or virtual textual/graphical annotations that are detached from entities, yielding the Entity Centric and Annotation Centric kinds. Each kind is further subdivided as follows in Table 11:
| Kind | Component Type | Definition |
| Entity Centric | ENTITY | Entity existence. Whether an entity composed of general, common-level concepts is present in the image — including specific persons, objects, phenomena, scenes, and backgrounds. Scene text and graphics attached to entities, as well as the overall style and genre of the image, are also included in this type. |
| Entity Centric | ENTITY_TRAIT | Entity traits. Objective, intrinsic properties of knowledge-intensive entities that remain invariant across viewpoints, external environments, and variant subtypes. These represent the characteristics that a T2I model must correctly and fully render — beyond merely generating a generic entity — in order for the depiction to be recognizable as that specific knowledge-intensive entity. Accordingly, when an entity is classified as knowledge-intensive, an ENTITY_TRAIT VQA item is added alongside the corresponding generic ENTITY existence item. |
| Entity Centric | ATTRIBUTE_BINDING | Attribute binding. An entity’s color, shape, size, material, state, and similar properties. |
| Entity Centric | ENTITY_RELATION | Inter-entity relations. Spatial positioning, physical interaction, and temporal ordering among entities. |
| Annotation Centric | ANNOTATION | Annotation content and form. Virtual textual or graphical annotations detached from entities, including text of all types, symbols, formulas, charts, figures, and diagrams. |
| Annotation Centric | ANNOTATION_BINDING | Entity–annotation binding. The directional or positional relationship between an annotation and its associated entity. |
| Annotation Centric | ANNOTATION_RELATION | Multi-annotation relations. Layout and logical relationships among multiple annotations. |
Table 11. Detailed definition of VQA Component. Based on semantics within prompt, vqa are first divided into two major kinds, and then further subdivided into 7 component types
Eval Type: Eval Type specifies whether a VQA item assesses purely visual rendering quality or world-knowledge grounding.
VISUAL: Pure visual rendering verification. The prompt already provides all information sufficient for visual rendering—including entity types, corresponding attribute states, and inter-entity interaction relationships—and the entities described are generic persons, objects, or scenes without associated world knowledge. Only basic rendering, composition, and layout capabilities are required; no world knowledge is needed.
KNOWLEDGE: World-knowledge grounding verification. Tasks that can only be completed correctly by drawing on the T2I model’s world knowledge—either by explicitly requiring entities whose depiction is directly tied to domain-specific world knowledge, or implicitly requiring operations such as knowledge-associated retrieval, state simulation, knowledge annotation, chart rendering, or scientific illustration.
Has Text Rendering: Has Text Rendering indicates whether a VQA item involves the assessment of text rendering and typesetting capabilities.
This field is set to true if:
(a) Within the Entity Centric kind, there exists a verification of scene text attached to an entity;
(b) Within the Annotation Centric kind, there exists a verification of virtual textual annotations detached from entities.
Otherwise, if the VQA item does not involve text rendering or typesetting in any form, the field is set to false. Notably, graphical elements in the scene and virtual graphical annotations are treated as visual rendering—they assess fundamentally different model capabilities from the highly sensitive text rendering and typesetting ability—and are therefore not classified as text rendering. In such cases, Has Text Rendering is likewise set to false.
Evaluability-Oriented Annotations
We augment each VQA item with evaluation-oriented auxiliary annotations, including:
Knowledge Hint: A credibility-oriented guidance field for world-knowledge assessment. It contains: the knowledge operation type assessed by the VQA item, background knowledge provided to the judge model, and detailed judgment criteria—all consolidated into a single field. When the Judge Type is OPTIONAL, the judgment criteria additionally enumerate all plausible outcome scenarios.
Visual Grounding: An optional visual anchoring field for knowledge-intensive entities. High-quality reference images are sourced from Wikimedia Commons, retrieved either automatically via agent-invoked tool calls or through manual curation.
Knowledge Hint and Visual Grounding serve complementary roles. The specific retrieval procedure is as follows:
When the VQA decomposition agent encounters information postdating its knowledge cutoff, uncertain domain-specific knowledge, complex physical or chemical mechanisms, or obscure historical details, it MUST invoke an internet search tool to consult encyclopedic or authoritative sources. After thoroughly understanding the relevant background, the agent extracts the entity’s visually discriminative features, core judgment logic, and equivalent variants, and supplements the corresponding VQA item’s knowledge_hint field with this information, ensuring the explanation is highly detailed and accurate.
When the content being assessed involves knowledge-intensive entities—such as specific paleontological fossils, particular spacecraft models, rare laboratory instruments, or region-specific traditional costumes—or contains information from 2024 onward, the agent is
required to invoke the custom image search tool wikimedia_image_search for visual grounding.
Up to three images are retrieved from the Wikimedia Commons API and passed into the context of the agent model (Gemini 3.1 Pro Preview), which is prompted to assess image quality—verifying that each image is sharp, complete, highly recognizable, and representative, and that it captures the typical intrinsic visual characteristics of the knowledge-intensive entity.
Upon confirmation, the image URL is inserted into the
visual_groundingfield of the corresponding VQA item, and theknowledge_hintfield is enriched based on the image content. If the retrieved images do not meet the requirements, the search query is revised and retried; a maximum of one retry is allowed.
Both
knowledge_hintandvisual_groundingare then subject to human review, with corrections applied as needed.
The positive contribution of these VQA-level knowledge auxiliary fields to judge performance has been validated through ablation studies, as detailed in § 4.2.
Examples
Table 12 presents a representative instance from the VQA judge protocol, shown with accompanying model generated image and visual grounding.
| Case ID | 943aca39-bd1d-5b93-accd-10943434b4bf | |
| Prompt | Draw a torii gate at the entrance of a traditional Japanese shrine. | |
| Category | 1. Human Civilization & Society | |
| Category | 2. Humanities & Architecture | |
| Category | 3. Architectural Styles | |
| Category | 4. Japanese Architecture and Key Components | |
| # | Question, Labels & VG. | Knowledge Hint |
| 1 | The image contains a gate-like architectural structure consisting of two upright pillars and a top crossbar. ENTITY VISUAL REQUIRED | Pure visual assessment. Irrelevant to the current world knowledge question, it confirms whether the basic entity of an architectural structure similar to an archway or a door frame exists in the image. |
Table 12. An example showcasing the VQA judge protocol, shown with accompanying model generated image and visual grounding. Demonstrates the composition of VQA Checklist, its internal logical relationships, and machine-scoring auxiliary information.
2 The gate-like architectural structure in the image conforms to the visual characteristics of a traditional Japanese torii gate.
ENTITY_TRAIT
KNOWLEDGE
REQUIRED
Figure 11.
Explicit knowledge recall verification. The typical structure of a torii gate includes two upright pillars and a top crossbar (the topmost being the "kasagi", often with a "nuki" passing through the pillars below it), and some have a short vertical wooden piece or plaque ("gakuzuka") between the two crossbars. The style could be a Myojin torii with the ends of the top crossbar curving upwards, or a Shinmei torii with a straight top. The most common color is vermilion (often with black caps or bases), or the natural colors of stone or wood. There may also be a shimenawa (sacred rope) or shide (paper streamers) hung on the crossbar. PASS if the architecture in the image possesses the above characteristics; FAIL if it is drawn as a Chinese-style archway (paifang, including multiple complex flying eaves and brackets, or multi-pillar structures like three doors and four pillars), features Chinese-style coiled dragon sculptures, a Western stone arch, or a simple modern door frame.
VG. Credits: Torii gate of Takekoma-jinja shrine.JPG by Bachstelze, CC BY-SA 3.0 https://creativecommons.org/licenses/by-sa/3.0>, via Wikimedia Commons.
Source: https://commons.wikimedia.org/wiki/File:Torii_gate_of_Takekoma-jinja_shrine.JPG
3 The torii gate is situated in an entrance environment leading to a sacred precinct, with the background containing elements such as a path and a natural forest or traditional wooden architecture.
KNOWLEDGE
REQUIRED
Implicit environmental knowledge reasoning. The prompt requires the torii gate to be at the entrance of a traditional Japanese shrine, marking the boundary between the secular world and the sacred precinct. PASS if the image shows the torii gate standing in front of a path (sando), and the surrounding environment has natural forests, stone lanterns, komainu (guardian lion-dogs), or faintly visible traditional Japanese building complexes deep in the background, which reflect the entrance characteristics of a sacred precinct; FAIL if the torii gate is placed right in the middle of an unrelated modern commercial street without any shrine background, placed indoors, or surrounded entirely by European/Chinese-style architecture that does not conform to the setting of a traditional Japanese shrine entrance.
4 The image contains additional text annotations explaining the name of the torii gate, the background of the shrine.
ANNOT
KNOWLEDGE
OPTIONAL
Optional knowledge annotation task. If there are no additional text annotations in the image (excluding the characters on the plaque inherent to the torii gate itself), evaluate as NA silently. If the model proactively generates introductory text, PASS if the text content is related to "Torii", "Shrine", etc., and is spelled correctly; FAIL if there are obvious spelling errors or if the annotations are inaccurate texts such as "Arch-way".
5 The additional text annotations in the image correctly correspond to concepts related to the torii gate or shrine.
ANNOT_BIND
KNOWLEDGE
OPTIONAL
Optional knowledge annotation task. If there are no additional text annotations in the image, evaluate as NA silently. If the model proactively generates text, PASS if the text content correctly describes the corresponding structure or overall concept of the torii gate, and logically matches the image elements; FAIL if the text cannot correspond to the architectural structure, resulting in a logical contradiction.
6 The central connection between the two horizontal beams of the torii gate is depicted as a short vertical wooden post, a rectangular plaque, or left completely empty.
ENTITY_TRAIT ADDITIONAL
Potential checkpoint for explicit knowledge recall. In a traditional Japanese torii gate, the exact center between the two horizontal beams (the upper kasagi/shimaki and the lower nuki) typically features a short vertical wooden post (gakuzuka), and sometimes a rectangular plaque (shingaku) bearing the shrine’s name is hung on it; in some minimalist styles (such as the shinmei torii), this position lacks any connecting components. PASS if the image depicts this area as a vertical wooden post, a rectangular plaque, or a completely empty space without any structure. FAIL if the model spontaneously generates bizarre wavy or twisted components, or decorations such as European wrought iron patterns or Chinese auspicious cloud/ruyi motifs that completely contradict the common knowledge of traditional Japanese architecture.
Human Ground-Truth Annotation for Judge Validation
Figure 7 shows the human ground-truth annotation interface used for judge reliability validation on the Skill-Tree Benchmark. Annotators review the generated image, the original prompt, and the structured VQA checklist, then assign ground-truth labels to each VQA item. These annotations are used to compute VQA-level agreement and aggregated case-level agreement between human annotators and the MLLM judge.

Figure 7. Human ground-truth annotation interface for judge reliability validation. Annotators label structured VQA items, which are used to compute VQA-level and case-level human–judge agreement.
Reliability of the human reference labels. To assess the consistency of the human references used for judge validation, five annotators independently labeled the Required VQAs in 50 sampled Skill-Tree cases, yielding 288 VQA units. We measure agreement among the five annotators using nominal Krippendorff’s , obtaining 0.766. Table 13 reports the distribution of vote patterns and the corresponding majority labels.
Table 13: Inter-annotator agreement on 288 Required VQA units from 50 sampled Skill-Tree cases. A vote pattern of 5–0, 4–1, or 3–2 denotes the number of annotators supporting the majority and minority labels, respectively; it does not specify whether the majority label is PASS or FAIL.
| Vote pattern | Count | Percentage | Majority PASS | Majority FAIL |
| 5–0 | 238 | 82.64% | 190 | 48 |
| 4–1 | 29 | 10.07% | 23 | 6 |
| 3–2 | 21 | 7.29% | 12 | 9 |
| Total | 288 | 100.00% | 225 | 63 |
Table 13.
Unanimous agreement is observed on 82.64% of the VQAs, and 92.71% receive agreement from at least four of the five annotators. Only 7.29% have a 3–2 split. These results support the consistency of the human references on the sampled cases. Inter-annotator agreement describes consistency among human labels; it is distinct from the human–judge agreement reported in § 4.2 and does not by itself establish the factual correctness of every reference label.
Human Evaluation Protocol for the Generalized Benchmark
Human-Machine Collaborative Evaluation Details
As described in § 3.2, the Generalized Benchmark adopts a human-judge collaborative protocol because open-ended world-knowledge-grounded generation requires both structured knowledge assessment and subjective human perception. This appendix provides the detailed human evaluation dimensions, scoring rubrics, quality-control procedure, and representative scoring examples.
We introduce a comprehensive evaluation encompassing six dimensions (Table 14) and design a human-judge collaborative evaluation framework to leverage the respective strengths of both Machine Evaluation and Human Evaluation.
Table 14: Definitions and boundaries of the evaluation dimensions. This taxonomy explicitly disentangles world-knowledge grounding from general semantic and visual metrics. Notably, overall satisfaction is not a linear aggregation of sub-dimensions, but a holistic, task-dependent judgment that captures complex trade-offs like factual vetoes and aesthetic compensation.
| Dimension | Evaluation Dimension | Definition |
| World-Knowledge Grounding | Knowledge Correctness | The degree to which core knowledge expressions are rendered correctly, reflecting the model’s understanding of the relevant world knowledge. |
| World-Knowledge Grounding | Knowledge Richness | The extent to which the model supplements core knowledge points with additional topically relevant knowledge, reflecting the depth and breadth of its world-knowledge understanding. |
| Semantic | Non-Knowledge Visual Instruction Following | The degree to which visual requirements specified in the prompt—beyond knowledge correctness and richness—are fulfilled; i.e., the semantic alignment between the image and the instruction. |
| Visual | Aesthetics | An assessment of the aesthetic quality of the generated image, including lighting, color, composition, and other aesthetic factors, reflecting the model’s capacity for visually appealing generation. |
| Visual | Image Quality | An assessment of visual quality, including sharpness, structural integrity, detail fidelity, and realism. Factual errors do not, by themselves, lower this score; a lower score requires an independently observable visual defect, such as blurring, structural distortions, or geometric interpretation errors. |
| Holistic | Overall Satisfaction | A holistic judgment representing the ultimate user acceptance of the generated image. Rather than a linear weighted average of other dimensions, it reflects a comprehensive, task-context-dependent utility assessment that accounts for complex cross-dimensional interactions |
Table 14.
Overall Satisfaction (OS) is treated as an independent holistic MOS rather than a fixed weighted sum of sub-dimensions. Its interpretation depends on the task context. In strongly knowledge-constrained tasks, such as scientific diagrams or historical/cultural depictions, errors in core knowledge can dominate the final judgment even when the image is visually appealing. In more open-ended or weakly knowledge-constrained tasks, strong image quality and aesthetics may partially compensate for minor non-core factual imperfections. Therefore, OS is scored as a task-dependent utility judgment that reflects the interaction among knowledge correctness, knowledge richness, visual instruction following, image quality, and aesthetics.
Machine-scored Non-knowledge Visual Instruction Following
For the Generalized Benchmark, non-knowledge visual instruction following is evaluated by an internal visual-instruction judge. This dimension is separated from world-knowledge grounding. because many prompts contain explicit visual requirements that do not require external knowledge, such as the number of subjects, viewpoint, layout, composition, color, style, and other directly specified visual constraints. These requirements affect the perceived success of a generation, but should not be conflated with knowledge correctness or knowledge richness.
Given a case , the judge decomposes the non-knowledge visual requirements into atomic visual units. Each unit is assigned a weight according to its importance in the prompt, and the judge predicts whether the unit is correctly satisfied by model . The raw instruction-following score is computed as a weighted response ratio:
Here, denotes the raw non-knowledge visual instruction-following score, is the number of decomposed atomic visual units, is the weight of the -th unit, and indicates whether this unit is correctly satisfied by model . The score is bounded in , where a larger value indicates better adherence to explicitly specified visual requirements.
To make this machine-scored dimension comparable with human-rated MOS dimensions, we calibrate the raw score to the same MOS scale:
This linear calibration maps to MOS 1 and to MOS 5. We use this calibrated score only as a comparable evaluation dimension in the human–judge collaborative protocol; it is not used as a fixed component in the overall satisfaction score. Overall satisfaction remains an independent holistic human MOS that reflects task-dependent trade-offs among knowledge correctness, knowledge richness, instruction following, aesthetics, image quality, and other perceptual factors.
Judge-derived MOS Conversion for Knowledge Scores
In the Generalized Benchmark, structured judge scores and human-rated MOS dimensions are reported on different numerical scales. The MLLM judge produces normalized scores in for knowledge correctness and knowledge richness, while human annotators provide MOS ratings on a discrete 1–5 scale. To make these dimensions comparable in joint analysis and visualization, we linearly convert the judge-derived knowledge scores to the MOS scale.
For a model and a generalized case , the judge-derived knowledge correctness score is computed from Required VQAs detailed in (2), while the knowledge richness score is computed from Optional VQAs detailed in (3).
We then convert each normalized judge score to a judge-derived MOS value using a linear mapping:
Thus, a score of 0 corresponds to MOS 1, and a score of 1 corresponds to MOS 5. This conversion does not change the ranking induced by the original judge scores; it only places judge-derived knowledge dimensions and human-rated dimensions on a common 1–5 scale.
For model-level reporting, we average the converted MOS values over all generalized cases:
Human-rated dimensions, including image quality, aesthetics, and overall satisfaction, are collected directly on the 1–5 MOS scale and averaged separately. We emphasize that the converted judge-derived MOS values are used only for scale alignment and joint analysis, not as substitutes for human MOS. In particular, overall satisfaction is kept as an independent human judgment rather than computed as a fixed weighted sum of knowledge correctness, knowledge richness, image quality, and aesthetics.
Human Evaluation Scoring Rubrics
Annotators assign an integer rating from 1 to 5 separately for image quality, aesthetics, and overall satisfaction, using the anchors in Table 15.
| Score | Image Quality | Aesthetics | Overall Satisfaction |
| 1 Bad | Severe blur, distortion, fragmentation, or structural collapse makes the main subjects or essential visual information unrecognizable. The image cannot support basic comprehension. | Severely disorganized composition, conflicting colors, or incoherent visual treatment. The image lacks a clear hierarchy and produces a strongly unfavorable visual impression. | The task is fundamentally unfulfilled. Severe core-knowledge errors, missing essential content, or overwhelming visual defects make the result unusable. |
| 2 Poor | Subjects remain recognizable, but substantial deformation, misalignment, interpenetration, or poorly rendered details seriously impair visual clarity and usability. | Weak composition or an unclear focal point, with distracting clutter, discordant colors, or inconsistent styling. These issues substantially impair visual appeal. | The result falls substantially short of the task. Major knowledge errors, missing supporting information, instruction violations, or poor visual execution substantially limit its usefulness. |
| 3 Fair | Main subjects and structures are discernible and broadly intact. Noticeable local artifacts, blur, repetition, or weak details remain, but the image is visually usable. | Acceptable composition and generally compatible colors and styling, but limited hierarchy, balance, or refinement. The overall appearance is adequate but unremarkable. | The result is usable but only partially fulfills the task. Knowledge coverage, instruction following, or visual presentation has noticeable limitations that reduce its overall usefulness. |
| 4 Good | Clear subjects and most details, with coherent structures and spatial relations. Only minor local artifacts or simplifications remain, without materially impairing comprehension. | Well-organized composition, a clear focal point, and harmonious visual treatment. Minor weaknesses in spacing, balance, or refinement do not undermine the favorable overall impression. | The task is largely fulfilled, with substantially correct core knowledge, sufficient supporting content, and good visual execution. Minor omissions or imperfections remain. |
| 5 Excellent | Crisp, complete rendering with stable details and coherent geometry. Textures and lighting are appropriate to the visual form, with no evident distracting artifacts. | Refined composition, clear visual hierarchy, and cohesive colors and styling. The presentation is highly effective and visually appealing for the intended task. | The task is fulfilled exceptionally well. Core knowledge is accurate, relevant supporting content is comprehensive, and instructions are faithfully followed, with strong visual execution and high overall usefulness. |
Table 15. Five-point rubrics for human evaluation. The descriptions provide rating anchors rather than a checklist whose conditions must all be satisfied. Overall satisfaction is rated independently and is not computed as a fixed weighted sum of the other dimensions.
Applying the dimension definitions. Ratings follow the dimension definitions in Table 14. For image quality, factual errors alone do not lower the rating; a lower rating requires an independently observable visual defect, such as blurring, structural distortion, or geometric interpenetration. Aesthetics assesses visual appeal and coherence within the intended visual form, while compliance with explicitly requested visual attributes is assessed under non-knowledge visual instruction following. Overall satisfaction remains a holistic, task-dependent judgment rather than a fixed weighted combination of dimension scores.
Task-dependent overall satisfaction. In strongly knowledge-constrained tasks, errors in core knowledge can dominate overall satisfaction even when the image is visually appealing. In more open-ended tasks, strong visual presentation may partially compensate for minor, non-core factual imperfections. Annotators therefore apply the overall-satisfaction anchors in the context of the task rather than mechanically averaging dimension scores.
Quality Control for Human MOS Evaluation
To ensure the stability and reproducibility of human MOS evaluation, we adopt a standardized annotation quality-control procedure. Before formal evaluation, annotators receive unified training on the task definition, evaluation dimensions, scoring rubrics, representative examples, and common boundary cases. A pilot annotation round is then conducted on a diverse subset of cases, followed by calibration discussions to resolve disagreements and refine boundary rules.
Annotators are admitted to the formal evaluation only after passing a qualification test against reference annotations.
Independent ratings. During formal evaluation, each image is initially rated by three annotators on image quality, aesthetics, and overall satisfaction, using the five-point rubrics in Table 15. Annotators complete their initial ratings independently and cannot see one another’s scores. The interface provides the structured judge evidence described in § 3.2; independence therefore refers to ratings across annotators rather than blinding to judge outputs.
Pre-adjudication agreement. We assess inter-annotator agreement using the initial ratings, before discrepancy review or adjudication. The agreement analysis yields Krippendorff’s of 0.7172. We additionally characterize disagreements using the range between the highest and lowest ratings assigned by the three annotators to the same image on the same dimension. Table 16 summarizes the distribution of initial rating patterns.
| Initial rating pattern | Score range | Percentage |
| Unanimous agreement | 0 | 57.02% |
| Adjacent-level disagreement | 1 | 30.17% |
| Larger disagreement | 12.81% |
Table 16. Distribution of initial human rating patterns before discrepancy review or adjudication. Score range denotes the difference between the highest and lowest scores assigned by the three annotators to the same image on the same dimension.
Discrepancy review and aggregation. The aggregation and review rules are specified before formal annotation. Unanimous ratings are accepted directly, and ratings with a range of one are aggregated according to the predefined rule. Only rating items with a difference of more than one point between the highest and lowest scores undergo review and adjudication. Thus, 87.19% of the evaluated rating items are processed without review or adjudication, whereas 12.81% undergo this process. Final MOS values are computed according to the predefined aggregation rule after resolving these larger disagreements. The reported agreement statistics are computed from the initial independent ratings and do not incorporate any changes made during review or adjudication.
Representative Scoring Example
Table 17 shows 4 representative Text-to-Image generation pattern with world-knowledge grounding using the same instruction:
Nano Banana 2
KC. 5: EXCELLENT
KR. 5: EXCELLENT
IF. 5: EXCELLENT
IQ. 5: EXCELLENT
VA. 5: EXCELLENT
OS. 5: EXCELLENT
Reason: The image perfectly renders the philosophical concept with rich content and impeccable composition, achieving a flawless balance of instruction following and top-tier visual quality. Overall satisfaction: 5.
Case B: Knowledge is correct, but knowledge expression is insufficient. Needs to improve visual rendering capability.
Recraft V4
KC. 5: EXCELLENT
KR. 4: GOOD
IF. 5: EXCELLENT
IQ. 4: GOOD
VA. 3: FAIR
OS. 4: GOOD
Reason: While the knowledge is accurate and instructions are fully followed, limited content richness and average visual aesthetics result in a relatively conventional presentation. Overall satisfaction: 4.
Case C: Knowledge contains critical errors, while aesthetics remain comparatively strong. Needs to improve knowledge grounding and reasoning capability.
Midjourney v7
KC. 1: BAD
KR. 2: POOR
IF. 2: POOR
IQ. 3: FAIR
VA. 4: GOOD
OS. 2: POOR
Reason: The image severely fails in theme-related instruction following yet stands out through its distinctive picture-book aesthetic and high-quality rendering, showcasing a sophisticated artistic sensibility. Overall satisfaction: 2.
Case D: Knowledge is largely correct, but basic visual instruction following is poor. Needs to visual instruction following capability.
FLUX.2 Pro
KC. 3: FAIR
KR. 3: FAIR
IF. 3: FAIR
IQ. 3: FAIR
VA. 2: POOR
OS. 3: FAIR
Reason: The core knowledge is rendered adequately, but the cluttered background elements and minor glyph imperfections undermine the overall visual quality. Overall satisfaction: 3.
Attribution Framework
The main experiments report model performance and key findings across the Skill-Tree Benchmark, the Generalized Benchmark, and the General-Domain T2I Benchmark. This appendix summarizes how these findings can be attributed through structured evaluation signals and case-level metadata. The goal is not to simply list failure cases, but to explain why models fail and where such failures are concentrated.
Two-Level Attribution Framework
We perform attribution at two levels. Signal-level attribution identifies the source of failure from intermediate evaluation signals, such as Required VQA failures, Optional VQA coverage, non-knowledge visual instruction following, annotation-related failures, and human MOS dimensions. Slice-level attribution then groups these signals by taxonomy and task metadata, such as L1–L4, education level, knowledge operation type, difficulty, and knowledge quantity.
Attribution on the Skill-Tree Benchmark
For the Skill-Tree Benchmark, attribution focuses on atomic knowledge-point mastery. We first identify low-performing L4 atomic knowledge points using case-level correctness. We then inspect failed Required VQAs and group them by component type, such as ENTITY, ENTITY_TRAIT, ATTRIBUTE_BINDING, ENTITY_RELATION, ANNOTATION, ANNOTATION_BINDING, and ANNOTATION_RELATION. Finally, we place these L4 points back under their corresponding L3 branches to distinguish local atomic knowledge gaps from broader branch-level weaknesses.
This process helps determine whether a model fails because it cannot recall the target knowledge concept, because it renders the relevant entity with incorrect attributes or relations, or because it fails to express the required knowledge through annotations, formulas, arrows, or text-image bindings.
Figure 8(a) provides the full Skill-Tree case-level heatmap across models, L1/L2 taxonomy slices, education levels, and knowledge operation types. This expanded view complements the compressed main-text visualization by showing how model performance varies across fine-grained knowledge domains and task metadata. The heatmap is intended for diagnostic inspection rather than for deriving a single global ranking: different models may exhibit different strengths across taxonomy branches, education levels, and knowledge operation types.
VQA-Level Component Failure Analysis
Case-level correctness identifies whether a Skill-Tree case succeeds or fails, but it does not reveal which part of the visual knowledge expression caused the failure. To localize failure sources, we further analyze VQA-level pass rates by component type. Each Required VQA is grouped according to its component label, including entity-level checks, attribute and binding checks, relation checks, and annotation-related checks. This allows us to distinguish failures caused by missing or incorrect entities from failures caused by incorrect relations, attribute binding, textual labels, formulas, arrows, or annotation–image bindings.
Figure 5(b) reports the VQA-level component heatmap, which localizes Skill-Tree failures beyond case-level correctness. We observe three main patterns.
First, entity-centered components generally obtain higher pass rates than relation- and annotation-centered components. This suggests that models often recognize the relevant visual subject, but this alone is insufficient for knowledge correctness. For example, a model may draw the correct object or scene while failing to express the required physical relation, structural constraint, or symbolic convention.
Second, annotation-related components form a major bottleneck. Components such as ANNOTATION, ANNOTATION_BINDING, and ANNOTATION_RELATION require text, formulas, arrows, labels, charts, or callouts to be both visually legible and semantically bound to the correct image regions. Failures in these components are especially harmful for world-knowledge-grounded generation, because many scientific, mathematical, and explanatory prompts rely on annotations as part of the knowledge expression rather than as decorative elements.
Third, model differences are more pronounced at the component level than at the entity level. Some models can pass many entity-related checks but still fail relation or annotation checks, indicating that their bottleneck lies not in generating relevant objects, but in organizing structured knowledge evidence. This supports the diagnostic value of our VQA decomposition: it separates entity generation failures, attribute-binding failures, relational reasoning failures, and annotation-rendering or annotation-binding failures, instead of collapsing them into a single case-level score.
Attribution on the Generalized Benchmark
For the Generalized Benchmark, attribution focuses on comprehensive failure decomposition in open-ended generation scenarios. For each case, we inspect judge-derived signals, including knowledge correctness, knowledge richness, and non-knowledge visual instruction following, together with human-rated MOS dimensions, including image quality, aesthetics, and overall satisfaction.
We then map the failure back to task metadata, including L3 branch, knowledge operation type, education level, difficulty, knowledge quantity, and involved L4 atomic knowledge points when available. This allows us to analyze whether low overall satisfaction is mainly caused by incorrect knowledge, incomplete knowledge expression, poor instruction following, weak image quality, or aesthetic issues, and whether such failures are concentrated in particular task slices.
Reporting Formats
Table 18 summarizes the common reporting formats used for attribution analysis. These formats are not intended to introduce additional metrics; instead, they organize existing evaluation signals into interpretable views for inspecting where and why models fail.
| Benchmark View | Reporting Format | Purpose |
| Skill-Tree | L1/L2/L3/L4 hierarchical table | Shows accuracy across domains, branches, and L4 atomic knowledge points. |
| Skill-Tree | VQA component heatmap | Identifies dominant failure types, such as entity, relation, annotation, or binding errors. |
| Skill-Tree | L4-under-L3 heatmap | Reveals which L4 atomic knowledge points fail within the same L3 branch. |
| Skill-Tree | Annotation-related failure table | Measures whether text, formulas, arrows, labels, or callouts form a bottleneck for knowledge expression. |
| Generalized | Dimension-wise score table | Reports knowledge correctness, knowledge richness, non-knowledge visual instruction following, and human MOS dimensions. |
| Generalized | Slice-based heatmap | Analyzes failures by L3 branch, difficulty, knowledge operation type, education level, or knowledge quantity. |
| Generalized | Dimension-vs-overall plot | Examines how sub-dimensions relate to overall satisfaction. |
| Generalized | Representative case card | Presents a multi-dimensional attribution path for individual cases. |
Table 18. Common reporting formats for attribution analysis.
Empirical Analysis of Automatic Evaluation Failures
Qualitative Analysis of Failure Patterns
Visual reasoning failures. The MLLM judge exhibits insufficient visual reasoning capability, manifesting in the following sub-types:
Incorrect reasoning about spatial orientation and coordinates involving human figures;
Misrecognition of textual and graphical content;
Errors in structural understanding.
Alignment failures. The judgment criteria of the MLLM judge deviate from human evaluation standards in two opposing directions:
Over-strict judgment, most notably a systematic tendency to penalize handwritten or rendered text content;
Over-lenient judgment, where the presence of duplicate or redundant elements is overlooked—the judge passes a criterion as long as at least one conforming instance is found, regardless of additional incorrect instances.
Representative failure cases are shown in Table H.1. For each case, we report the issue category, the VQA question, the judge model’s reasoning, and the identified failure type.
| Samples | VQA, Reasoning, and Issue |
Table H.1. Failure Patterns of automatic evaluation system It has been shown that the MLLM’s visual perception and reasoning performance significantly impact the evaluation of Text-to-Image (T2I) generation.
Figure 12.
ID: c8ae24b5-f58c-549a-8882-2638730408d0
VQA: The bride’s back is facing the group of women.
Reasoning: In the image, the bride wearing a white wedding dress has her back to the audience, while multiple women in the background are facing toward the bride, clearly demonstrating the spatial relationship of the bride standing with her back to the crowd. The pass condition is satisfied.
Issue: Misidentification of real-world viewers as in-scene audience; confusion of spatial relationships.
Figure 13.
ID: 0b4b9e8f-e4e7-5265-a3d7-92576e68347d
VQA: The annotations for the foci in the image accurately provide the coordinate values and , and the corresponding marker points are located between the values 2 and 3 (and between -2 and -3) on the X-axis.
Reasoning: As an optional component, the focus annotations provide accurate coordinate values and . In the image, the solid markers representing these two foci indeed lie on the x-axis within the grid intervals between 2 and 3 (closer to 2) and between -2 and -3 (closer to -2), respectively, corresponding to the precise positions of approximately . The points are fully consistent with the given values, satisfying the acceptance criteria.
Issue: Error in spatial coordinate identification/inference of the annotation.
Figure 14.
ID: f16b8fe9-035d-5780-89bf-0cc538e22d9f
VQA: There are additional text annotations in the image.
Reasoning: As an optional element, the model generated textual annotations on the nautical chart held by the character. However, upon closer inspection, the text—composed of Latin-like characters (e.g., “A U O D P M”, “O R Y M I”), is nonsensical when taken together and forms unintelligible pseudo-gibberish. Therefore, it is judged as a failure.
Issue: The automated evaluation is overly strict. It incorrectly treats decorative or visually embedded content attached to an object as meaningful annotation text, and the criteria for assessing written content are excessively harsh.
Quantitative Breakdown of Failure Modes
To complement the qualitative case analysis above, we provide a quantitative breakdown of the MLLM judge’s remaining failure modes. Figure 9(a) reports VQA-level human–judge agreement across taxonomy slices and VQA categories, while Figure 9(b) lists the top-10 lower-agreement cells. Overall, agreement remains high across most slices, supporting the use of the structured judge for large-scale batch evaluation.

Figure 9. Judge reliability analysis. (a) VQA-level human–judge agreement across taxonomy slices and VQA categories. (b) Top-10 lower-agreement slices.
The lower-agreement cells are not uniformly distributed. The lowest agreement appears in General Social Knowledge with ENTITY_TRAIT agreement of 0.71. Other low-agreement cells are mainly associated with relation-, binding-, and annotation-related VQA categories, including ENTITY_RELATION, ATTRIBUTE_BINDING, ANNOTATION_BINDING, and ANNOTATION_RELATION. These results indicate that the remaining judge instability is concentrated in fine-grained attribute recognition and structured visual reasoning, rather than uniformly affecting all VQA categories.
Relation-sensitive judgments. The overly lenient judgments described above motivate closer examination of relation-sensitive checks. In the evaluated ENTITY_RELATION slice, 774 VQAs account for 12.66% of all VQAs. Agreement with human annotations reaches 89.28% accuracy and Cohen’s . These statistics characterize the component as a whole; they do not isolate the frequency of errors involving exact counts, duplicates, or inconsistent instances. Such failure-specific effects require a targeted analysis.
Impact of Judge Errors on Model Rankings
The preceding qualitative and component-level analyses identify where the judge makes errors, but do not establish how those errors affect comparative model rankings. We therefore conduct a human-correction audit of a total of 50 prompt–image pairs sampled from the Skill-Tree outputs of five T2I models. We correct the identified judge errors and recompute the model rankings within the audited subset.
As shown in Table 20, Nano Banana 2 remains first, the fourth and fifth positions remain unchanged, and only the second and third positions are exchanged. The rankings before and after correction yield Kendall’s and Spearman’s . No model moves by more than one position. Within this audited subset, the corrections preserve the leading model and produce only a local change in ordering. The audit does not establish that judge errors are negligible across the full benchmark or within every task category.
| T2I model | Before | After | Absolute shift |
| Nano Banana 2 | 1 | 1 | 0 |
| Seedream 5.0 Lite | 2 | 3 | 1 |
| Qwen Image 2.0 | 3 | 2 | 1 |
| FLUX.2 Pro | 4 | 4 | 0 |
| Kling Image 3.0 Omni | 5 | 5 | 0 |
Table 20. Model rankings before and after human correction of identified judge errors in the five-model audit. These ranks are computed within the audited subset and should not be equated with the ten-model rankings in Table 24.
This analysis concerns the combined effect of the judge errors identified in the audit. It does not isolate the effect of overly lenient PASS decisions from other failure types. In particular, the component-level agreement statistics in H.2 should not be interpreted as a direct measurement of ranking changes caused specifically by duplicate-sensitive errors.
Future refinement. The limited ranking changes observed in the audit do not remove the need to address the failure patterns identified above. In particular, overly lenient judgments motivate explicit checks for object counts, duplicates, consistency across instances, and subject–attribute bindings when required by the task. We plan to evaluate whether these refinements reduce false-PASS errors while avoiding additional false-FAIL errors, and to explore human review for ambiguous cases. These refinements have not been incorporated into the protocol used for the reported results.
Additional Experimental Results
Complementary Results on the General-Domain T2I Benchmark
The General-Domain T2I Benchmark is used as a supplementary control to distinguish world-knowledge grounding from general image-generation capability. Unlike the Skill-Tree and General-ized benchmarks, this benchmark focuses on non-knowledge-intensive prompts and evaluates models along general instruction following, visual quality, and aesthetic quality. Its purpose is not to rank models by general visual performance, but to contextualize whether low world-knowledge scores can be explained by broad deficiencies in basic T2I generation.
Figure 10 reports the General-Domain T2I results. Several models achieve competitive general-domain scores, especially on instruction following, visual quality, or aesthetics, while still showing larger differences on Skill-Tree correctness and Generalized knowledge dimensions. This indicates that general T2I capability and world-knowledge grounding are related but not interchangeable.
Figure 10. Complementary evaluation on the General-Domain T2I Benchmark. We compare models on instruction following, visual quality, and aesthetic quality to contextualize world-knowledge grounding results.
For example, Qwen-Image 2.0 and FLUX.2 Pro perform well on general instruction following and visual quality, but still lag behind the strongest model in world-knowledge grounding. Midjourney V7 shows aesthetic strength, yet its knowledge correctness and richness remain weaker on the world-knowledge benchmarks. Kling Image 3.0 Omni remains within a comparable range on general-domain performance, suggesting that its weaker world-knowledge results should be interpreted more as a knowledge-specific limitation than as a broad deficiency in basic generation capability.
Overall, this supplementary benchmark supports the need for KnowVis: evaluating general instruction following, visual quality, or aesthetics alone cannot reliably reveal whether a model correctly grounds world knowledge in generated images.
Default-Judge Reliability and Component Ablations
We examine the contribution of structured auxiliary fields using Gemini 3.1 Pro Preview on all 480 annotated images in the Skill-Tree subset, including both the development and validation splits. We separately remove taxonomy context (category), knowledge hints (knowledge_hint), and visual grounding cues (visual_grounding), and also evaluate a setting without all three fields. Models, samples, VQAs, scoring rules, and aggregation are held fixed. Table 21 reports the complete ablation results.
| Setting | Cat. | KH. | VG. | Case-level Agreement Acc. | Case-level Agreement Rec. | Case-level Agreement Prec. | VQA-level Agreement Acc. | VQA-level Agreement Rec. | VQA-level Agreement Prec. |
| full protocol | ✓ | ✓ | ✓ | 88.12 | 85.19 | 95.11 | 94.51 | 80.22 | 92.07 |
| w/o category | ✗ | ✓ | ✓ | 86.46 (-1.66) | 85.86 (+0.67) | 91.73 (-3.38) | 93.03 (-1.48) | 77.44 (-2.78) | 87.29 (-4.78) |
| w/o knowledge hint | ✓ | ✗ | ✓ | 86.67 (-1.45) | 86.53 (+1.34) | 91.46 (-3.65) | 92.53 (-1.98) | 77.44 (-2.78) | 84.93 (-7.14) |
| w/o visual grounding | ✓ | ✓ | ✗ | 86.07 (-2.05) | 81.82 (-3.37) | 94.92 (-0.19) | 93.56 (-0.95) | 76.17 (-4.05) | 91.07 (-1.00) |
| w/o all components | ✗ | ✗ | ✗ | 86.04 (-2.08) | 83.84 (-1.35) | 92.91 (-2.20) | 92.49 (-2.02) | 76.89 (-3.33) | 85.21 (-6.86) |
Table 21. Judge reliability and component ablation on the Skill-Tree Benchmark. We report human-judge agreement at both the case level and the VQA level. Values are percentages; values in parentheses indicate changes in percentage points compared with the full protocol.
Removing any auxiliary field lowers case-level and VQA-level accuracy. Removing all three fields decreases accuracy by 2.08 percentage points at the case level and 2.02 points at the VQA level. The largest VQA-level precision reduction occurs when knowledge hints are removed, from 92.07% to 84.93%, a decrease of 7.14 points. These results show that explicit auxiliary information improves agreement with human annotations under the tested configuration. They do not establish that the judge is independent of its own world knowledge, visual perception, or reasoning capabilities.
Human–Judge Agreement on the Generalized Benchmark
The Skill-Tree validation focuses on bounded atomic knowledge tasks. We additionally assess the default Gemini 3.1 Pro Preview judge on 50 sampled Generalized cases to examine agreement in open-ended scenarios. Human review covers both Required VQAs, which assess core knowledge correctness, and Optional VQAs, which assess predefined supplementary knowledge expressions. We compare judge predictions with the human annotations at the VQA level, rather than comparing only the final case-level scores.
Table 22 shows high agreement with human annotations for both Required and Optional VQAs on the sampled cases. This analysis complements the Skill-Tree validation by directly examining open-ended knowledge expressions. The two studies use different evaluation samples, so their accuracy values should not be interpreted as a controlled comparison of evaluation difficulty. The present analysis concerns the default judge and does not establish cross-judge robustness on the Generalized Benchmark.
| VQA subset | Accuracy (%) |
| Required | 98.41 |
| Optional | 98.53 |
| All VQAs | 98.44 |
Table 22. VQA-level accuracy of the default judge against human annotations on 50 sampled Generalized cases. Required and Optional VQAs correspond to knowledge correctness and predefined knowledge richness, respectively.
Human–Judge Agreement Across Judge Settings
We compare four evaluation settings: the original Gemini 3.1 Pro Preview judge, a same-setting rerun, Gemini 3.5 Flash, and GPT 5.5. We report agreement with human annotations at both the VQA and case levels. VQA-level metrics assess individual checklist judgments, while case-level metrics assess the aggregated Required-VQA pass/fail decisions. Table 23 extends the main-text comparison with complete case-level results.
| Metric | Original | Rerun | Flash | GPT 5.5 |
| VQA Acc. | 94.51 | 91.74 | 92.21 | 91.33 |
| VQA Prec. | 92.07 | 83.63 | 86.74 | 86.03 |
| VQA Rec. | 80.22 | 74.44 | 73.41 | 69.11 |
| VQA | 0.8236 | 0.7366 | 0.7475 | 0.7140 |
| Case Acc. | 88.12 | 83.54 | 83.33 | 84.79 |
| Case Prec. | 95.11 | 91.29 | 92.89 | 93.41 |
| Case Rec. | 85.19 | 81.14 | 79.12 | 81.14 |
| Case | 0.7562 | 0.6628 | 0.6623 | 0.6903 |
Table 23. Complete human–judge agreement across four judge settings. Acc., Prec., and Rec. denote accuracy, precision, and recall, reported as percentages. denotes Cohen’s . Original and rerun refer to Gemini 3.1 Pro Preview; Flash refers to Gemini 3.5 Flash.
Across all tested settings, VQA-level accuracy remains above 91%. Although the rerun and alternative judges yield lower agreement metrics than the original evaluation, they retain substantial agreement with human annotations: VQA-level Cohen’s ranges from 0.7140 to 0.7475, while case-level ranges from 0.6623 to 0.6903. Their accuracy ranges from 91.33% to 92.21% at the VQA level and from 83.33% to 84.79% at the case level.
The same-setting rerun yields VQA-level and case-level of 0.7366 and 0.6628, respectively, compared with 0.8236 and 0.7562 in the original evaluation. These results show that agreement with human annotations remains substantial at both levels, while its exact magnitude varies across evaluations. The present comparison does not establish the cause of this variation. Appendix § 16.§ 5 further examines whether the main model-ranking trends remain stable across these judge settings.
Model Ranking Consistency Across Judge Settings
We compare the rankings of ten T2I models under the four judge settings, using the original Gemini 3.1 Pro Preview ranking as the reference. Table 24 provides the complete rankings and shifts relative to the baseline, and Table 25 summarizes their consistency. The original ranking is the reference ordering for this comparison, not an independently established ground-truth ranking.
| T2I model | Baseline rank | Gemini 3.1 Pro Preview (rerun) Rank | Gemini 3.1 Pro Preview (rerun) | Gemini 3.5 Flash Rank | Gemini 3.5 Flash | GPT 5.5 Rank | GPT 5.5 |
| Nano Banana 2 | 1 | 1 | 0 | 1 | 0 | 1 | 0 |
| Seedream 5.0 Lite | 2 | 2 | 0 | 2 | 0 | 2 | 0 |
| Recraft V4 | 3 | 4 | −1 | 3 | 0 | 3 | 0 |
| HunyuanImage 3.0 | 4 | 5 | −1 | 5 | −1 | 5 | −1 |
| Qwen Image 2.0 | 5 | 3 | +2 | 4 | +1 | 4 | +1 |
| Wan 2.7 Pro | 6 | 8 | −2 | 8 | −2 | 8 | −2 |
| FLUX.2 Pro | 7 | 6 | +1 | 6 | +1 | 7 | 0 |
| Seedream 4.5 | 8 | 7 | +1 | 7 | +1 | 6 | +2 |
| Kling Image 3.0 Omni | 9 | 9 | 0 | 10 | −1 | 9 | 0 |
| Midjourney V7 | 10 | 10 | 0 | 9 | +1 | 10 | 0 |
Table 24. Ranking shifts across judge settings for ten T2I models. The baseline is the original Gemini 3.1 Pro Preview evaluation; the Gemini 3.1 Pro Preview column reports the same-setting rerun. Rank 1 is best. We define , so positive values indicate an improvement in rank and negative values indicate a decline.
| Judge setting | Kendall’s | Spearman’s | Top-3 Jaccard | Max. shift |
| Gemini 3.1 Pro Preview (rerun) | 0.8222 | 0.9273 | 0.5000 | 2 |
| Gemini 3.5 Flash | 0.8222 | 0.9394 | 1.0000 | 2 |
| GPT 5.5 | 0.8222 | 0.9394 | 1.0000 | 2 |
Table 25. Ranking consistency relative to the original Gemini 3.1 Pro Preview ordering. Top-3 Jaccard measures overlap of the top-three model sets and does not measure their internal ordering. Maximum shift is the largest absolute rank difference among the ten models.
All three comparisons yield Kendall’s = 0.8222, while Spearman’s ranges from 0.9273 to 0.9394. The first two models remain unchanged in every setting. Gemini 3.5 Flash and GPT 5.5 also preserve the original top-three ordering. In the Gemini rerun, Qwen Image 2.0 replaces Recraft V4 in third place, giving a top-three set overlap of two models and a Jaccard index of 2/4 = 0.5. No model moves by more than two positions.
These results support the stability of the main ranking trends under the tested strong MLLM judges, while allowing local changes in ordering. They do not establish that absolute benchmark scores are invariant, that small score differences are statistically significant, or that smaller judge models would provide comparable results. The distinct question of how identified judge errors affect rankings after human correction is examined in Appendix § 15.3.
Limitation
Although this paper systematically explores task definition, benchmark construction, and evaluation protocol design, several limitations remain.
First, the modeling of knowledge richness requires further refinement. Our current approach primarily uses optional knowledge expressions to assess whether generated images convey knowledge more completely. However, how to integrate predefined knowledge expressions with valid additional knowledge expressed in generated outputs, and how to aggregate them in a stable and generalizable manner, remain open questions.
Second, although the scoring protocol separates knowledge correctness from image quality, residual perceptual coupling may remain in human evaluation. Factual or semantic inconsistencies may influence perceived image quality even when annotators are instructed to assess these dimensions separately. Future versions will investigate a more explicit distinction among narrow image quality, visual/structural plausibility, and knowledge plausibility, and evaluate whether this refinement improves inter-annotator agreement and error attribution. This finer-grained separation has not been implemented as a separate set of reported evaluation dimensions in the current benchmark.
Third, evaluation reliability depends on the capabilities of the MLLM judges. Although the protocol can be implemented with different judge backends, its reliability still depends on each judge’s world knowledge, visual perception, and reasoning capabilities. Knowledge hints and visual grounding cues provide explicit references but do not eliminate these dependencies or potential model-specific biases. Our cross-judge experiments support the stability of the main ranking trends among the tested strong MLLMs, rather than across arbitrary judge models. Preliminary exploration with smaller models has not established comparable reliability, and their suitability as substitutes for the current judges remains an open question. Developing reliable evaluation with less capable and more accessible judge models is therefore an important direction for future work.
Finally, this paper primarily focuses on static image generation and has not yet been systematically extended to more complex scenarios such as image editing, video generation, or multi-turn interactive generation. These tasks introduce additional challenges, including temporal consistency, cross-turn knowledge preservation, multimodal constraint alignment, and local editing consistency. Extending the benchmark and evaluation protocol to address these challenges remains an important direction for future work.
Resources and Reproducibility
Experimental Setup Details
This section provides additional details about the evaluation sets, data splits, and model selection used in our experiments. We separate these details from the main text to keep the experimental section concise while preserving reproducibility.
Evaluation Sets
We use three evaluation sets with complementary roles: the Skill-Tree Benchmark, the Generalized Benchmark, and a General-Domain T2I Benchmark. Table 26 summarizes their full size, experimental subset size, coverage principle.
| Evaluation Set | Full / Eval. | Release | Coverage |
| Skill-Tree Benchmark | 1,622 / 480 | Full benchmark | All L4 atomic knowledge points |
| Generalized Benchmark | 550 / 100 | Full benchmark | All L2 subjects and L3 branches |
| General-Domain T2I Benchmark | 600 / 100 | 100-case subset only | General entities and scenes |
Table 26. Evaluation sets used in the experiments.
Skill-Tree Benchmark. The full Skill-Tree Benchmark contains 1,622 cases covering 240 L4 atomic knowledge points. For experiments related to judge development, reliability validation, ablation, and metric comparison, we uniformly sample 480 cases from the full benchmark, ensuring coverage of all L4 atomic knowledge points. This subset is further split into 240 visible development cases and 240 unseen validation cases.
Generalized Benchmark. The full Generalized Benchmark contains 550 open-ended cases. For model evaluation, we uniformly sample 100 cases from the full set, ensuring coverage of all L2 subjects and L3 branches. This subset is used to evaluate comprehensive world-knowledge grounding under complex open-ended scenarios. The MLLM judge evaluates structured dimensions, including knowledge correctness, knowledge richness, and non-knowledge visual instruction following, while human annotators rate image quality, aesthetics, and overall satisfaction using MOS.
General-Domain T2I Benchmark. The General-Domain T2I Benchmark contains 100 general generation tasks. It covers common entity and scene categories, including portraits, animals, objects, plants, and historical and cultural scenes. It is used as a supplementary evaluation of general T2I capability. Its instruction-following dimension reuses a mature internal automatic evaluation system, while image quality and aesthetics are evaluated using human MOS.
Data Splits
Skill-Tree split. The 480-case Skill-Tree experimental subset is sampled from the full 1,622-case Skill-Tree Benchmark with coverage over all L4 atomic knowledge points. The visible development set contains 240 cases and is used to inspect judge behavior, refine the MLLM-as-a-judge prompt, calibrate VQA construction guidelines, and iterate on the evaluation scheme. The unseen validation set contains another 240 cases and is used for model evaluation and for the ranking comparison across judges. Human–judge agreement and cross-judge agreement use the full 480-case subset.
Agreement Subsets
Human–judge agreement, human–human agreement, and cross-judge agreement use the subsets in Table 27. These subsets are taken from the experimental sets in Table 26.
| Check | Source | Subset |
| Human–judge agreement | Skill-Tree 480-case subset | All 480 cases |
| Human–judge agreement | Generalized 100-case subset | 50 sampled cases |
| Human–human agreement | Skill-Tree 480-case subset | 50 sampled cases |
| Cross-judge agreement | Skill-Tree 480-case subset | All 480 cases |
| Ranking consistency | Skill-Tree unseen validation split | All 240 cases |
Table 27. Subsets used for human–judge, human–human, and cross-judge agreement.
T2I Model Selection
We evaluate models from three overlapping groups: general-purpose T2I models, aesthetics- or visual-quality-oriented models, and text-/knowledge-oriented models. These groups are designed to test whether world-knowledge grounding aligns with general image quality, aesthetics, text rendering, or knowledge-oriented generation capability. Since these capabilities are not mutually exclusive, some models appear in multiple groups. Table 28 summarizes the model groups and provides references to the corresponding technical reports or official model announcements.
| Capability Axis | Representative Models |
| General-purpose T2I | Nano Banana 2 [35]; FLUX.2 Pro [36]; Recraft V4 [37]; Seedream 4.5 [5]; Hunyuan Image 3.0 [38]; Kling Image 3.0 Omni [39] |
| Aesthetics / Visual quality | Midjourney V7 [40]; Kling Image 3.0 Omni [39]; Seedream 4.5 [5] |
| Text / Knowledge | Nano Banana 2 [35]; Seedream 5.0 Lite [41]; Wan 2.7 Pro [42]; Qwen-Image 2.0 [6, 43] |
Table 28. Model groups used in the experiments. Groups are overlapping by design.
T2I Generation Settings
Prompt language. We use Chinese prompts for models developed in China and English prompts for models developed outside China. Each model uses a fixed prompt language throughout the evaluation. The bilingual prompts preserve the same underlying task and semantic intent.
Output resolution. We select 2K output whenever supported by the selected model variant; otherwise, we use the highest available output resolution. Table 29 summarizes the prompt language and output image dimensions for each model. All images use a square aspect ratio.
| Model | Provider | Language | Output resolution |
| Qwen-Image 2.0 | Alibaba | Chinese | |
| Wan 2.7 Pro | Alibaba | Chinese | |
| FLUX.2 Pro | Black Forest Labs | English | |
| Seedream 4.5 | ByteDance | Chinese | |
| Seedream 5.0 Lite | ByteDance | Chinese | |
| Nano Banana 2 | English | ||
| Kling Image 3.0 Omni | KlingAI | Chinese | |
| Midjourney V7 | Midjourney | English | |
| Recraft V4 | Recraft | English | |
| Hunyuan Image 3.0 | Tencent | Chinese |
Table 29. Providers, prompt languages, and output resolutions of the evaluated T2I models, ordered alphabetically by provider. We prioritize 2K generation and otherwise use the highest available resolution.
Model-specific settings. Nano Banana 2 is used with web search and image search enabled. Seedream 4.5 and Seedream 5.0 Lite are accessed through the Jimeng web interface. Qwen-Image 2.0 is accessed through chat.qwen.ai with Qwen 3.6 Plus selected as the conversational interface. Midjourney uses version 7 (–v 7). For Recraft, we use the V4 Standard variant.
Collection period. The image files used for evaluation were archived in April 2026.
Evaluation Consistency
To ensure fair and reproducible comparison, all models are evaluated on the same case sets, semantic prompt intents, benchmark metadata, scoring rubrics, and MLLM-as-a-Judge protocol within each evaluation set. Because each case includes bilingual prompts, the selected language variant is kept fixed according to the model-specific language setting used in the experiments. Unless otherwise specified, automatic scores are produced by the same MLLM judge under the same judge prompts, auxiliary metadata, and inference configuration.
Statistical Significance
All quantitative evaluation results reported in this paper are accompanied by 95% confidence intervals (CIs), computed via bootstrap resampling (1,000 resampling iterations) over the test set. The CI bounds reflect variability due to random sampling of test examples, and are reported as asymmetric intervals to avoid out-of-range values. Error bars in all figures and tables correspond to these bootstrap 95% CIs.
Released Resources
We release the benchmark schema, case metadata, bilingual prompts, prompt templates, VQA checklists, evaluation scripts, and reproducibility instructions.
Code. Please refer to https://github.com/QuantumWhisper42/KnowVis for complete codebase.
Dataset. Please refer to https://huggingface.co/datasets/QuantumWhisper42/KnowVis for complete datasets.
Model API Configuration
Dataset construction, checklist decomposition, labeling, and the default judge use Google Gemini models through the Vertex AI API (https://cloud.google.com/products/gemini-enterprise-agent-platform). Table 30 lists those settings. Table 31 lists Gemini 3.5 Flash and GPT 5.5, the two additional judges used for the cross-judge checks. Gemini 3.1 Pro Preview is unchanged from the Judge column of Table 30.
| Configurations | Dataset Construction | Automatic Evaluation | Misc | ||
| Generation | Filtering | VQA | Judge | Labeling | |
| API Entrypoint⋆ | 3.1-pro | 3-flash | 3.1-pro-customtools | 3.1-pro | 3-flash |
| Thinking Level | HIGH | UNSPECIFIED | |||
| Media Resolution | HIGH | N/A⋆⋆ | |||
| Built-in Tools | Grounding with Google Search & URL Context | -† | |||
| Custom Tools | - | - | - | - | Wikimedia Commons ‡ |
Table 30. Google Gemini API configurations for dataset construction, checklist decomposition, labeling, and the default judge.
| Configuration | Gemini 3.5 Flash | GPT 5.5 |
| Model | gemini-3.5-flash | gpt-5.5 |
| Thinking / reasoning | HIGH | reasoning effort high |
| Media resolution | HIGH | not set |
| Built-in tools | Google Search and URL Context | none |
| Structured output | same as the default judge | native |
Table 31. Additional judges used for the cross-judge checks. Gemini 3.5 Flash reuses the default Judge settings in Table 30.
Local Computing Resources
We used a MacBook Pro equipped with an Apple M3 Pro chip and 36 GB of unified memory to compute CLIPScore and VQAScore. Due to memory constraints, CLIP-FlanT5-XL was used to compute VQAScore.
Broader Impacts
KnowVis is an evaluation benchmark and does not introduce or deploy a new generative model. Its potential positive impacts include improving the factual reliability of text-to-image generation, supporting educational and scientific visualization, and providing the community with a structured diagnostic resource for evaluating world-knowledge grounding. By decomposing knowledge constraints into VQA-based evaluation units, the benchmark can help identify whether model failures arise from knowledge recall, reasoning, annotation errors, visual instruction following, or general image quality issues.
Potential negative impacts are mainly indirect. The benchmark and its feedback could be misused to optimize models for generating more convincing but misleading knowledge-intensive images, such
as incorrect educational diagrams, deceptive scientific illustrations, or fabricated professional-domain content. In addition, over-reliance on automated MLLM-as-a-judge evaluation may propagate judge errors or biases, especially in high-stakes, time-sensitive, or professional domains.
To mitigate these risks, we recommend releasing the benchmark with clear documentation, intended-use statements, and known limitations. For time-sensitive or professional-domain cases, future versions should include explicit temporal references, source verification, and update mechanisms where appropriate. Human review should remain necessary for high-stakes applications, and MLLM-as-a-judge results should be interpreted as diagnostic signals rather than definitive judgments.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Answer: [Yes]
The Abstract and Introduction 1 clearly state the paper’s three contributions and scope. Experimental results are consistent with the claimed goals, demonstrating that KnowVis reveals structural weaknesses of existing T2I models in world-knowledge grounding.
Guidelines:
The answer [N/A] means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper includes a dedicated Limitation J section covering three aspects: knowledge richness metric refinement, coupled evaluation dimensions in open-ended scenarios, and the restriction to static image generation. The key assumption of MLLM judge reliability is addressed through validation against human annotations.
Guidelines:
The answer [N/A] means that the paper has no limitation while the answer [No] means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate “Limitations” section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [N/A]
Justification: This paper presents a benchmark and evaluation protocol without theoretical results, proofs, or formal theorems.
Guidelines:
The answer [N/A] means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The automated data construction and filtering pipeline, benchmark data, and code are all publicly released, along with API configurations, system prompts, and other implementation details. Full reproducibility information is consolidated in Appendix § 18.
Guidelines:
The answer [N/A] means that the paper does not include experiments.
If the paper includes experiments, a [No] answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in
some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The dataset and code are publicly released with sufficient instructions for reproducing all main experimental results, as detailed in Appendix § 18.
Guidelines:
The answer [N/A] means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so [No] is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results?
Answer: [Yes]
Justification: All experimental details including evaluated models, API configurations, and evaluation protocol settings are specified in the paper and Appendix § 18.
Guidelines:
The answer [N/A] means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Confidence intervals for all main evaluation results are reported using bootstrap resampling (1,000 iterations, 95% CI). No parametric distributional assumptions (e.g., normality) are made, as MOS-style scores derived from human judgments on text-to-image generation tasks do not follow a well-defined parametric distribution. For details, refer to Appendix § 18 Subsection K.1.7.
Guidelines:
The answer [N/A] means that the paper does not include experiments.
The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).
If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: The experiments primarily rely on API calls to commercial T2I and MLLM models, with no large-scale local compute required. Local computation is limited to running CLIPScore and VQAScore, which require only standard CPU/GPU resources. Detailed API usage and local computation resources is described in Appendix K.
Guidelines:
The answer [N/A] means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper).
Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?
Answer: [Yes]
Justification: The research conforms to the NeurIPS Code of Ethics. The benchmark data does not involve personal privacy. Human annotators were compensated at or above the local minimum hourly wage. The benchmark covers diverse knowledge domains to mitigate cultural bias, and potential misuse and judge-related risks are discussed in Appendix L.
Guidelines:
The answer [N/A] means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer [No], they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Broader impacts including positive impacts, potential negative impacts, and mitigation strategies are discussed in Appendix § 19.
Guidelines:
The answer [N/A] means that there is no societal impact of the work performed.
If the authors answer [N/A] or [No], they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: Portions of the dataset contain prompts generated by LLMs and some fields sourced from Wikimedia Commons under open licenses (CC and public domain), both have undergone human review to ensure quality and safety, discussed in Appendix § 10.1. We do not release a new image-generation model, and the benchmark does not include high-risk personal data.
Guidelines:
The answer [N/A] means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes]
Justification: All evaluated T2I models are properly cited. Wikimedia Commons assets are used under CC and public domain licenses, with detailed per-asset attribution (title, author, source, and license) included in the released dataset on Hugging Face, as described in Appendix K.
Guidelines:
The answer [N/A] means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset’s creators.
New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The released dataset and code are accompanied by documentation on Hugging Face and GitHub, covering dataset structure, field descriptions, license, and usage instructions, as described in Appendix K.
Guidelines:
The answer [N/A] means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [N/A]
Justification: The paper does not involve crowdsourcing or research with human subjects. Human evaluation was conducted by internal annotators and is not the subject of study.
Guidelines:
The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [N/A]
Justification: The paper does not involve crowdsourcing or research with human subjects, and therefore does not require IRB approval.
Guidelines:
The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: LLMs are core components of this work in two respects: as evaluated subjects (T2I-related models assessed by the benchmark) and as the backbone of the MLLM-based VQA judge protocol for automatic evaluation. Both usages are described in detail in the experimental setup and evaluation protocol sections.
Guidelines:
The answer [N/A] means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.
References
- [1]Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.arxiv.org/abs/2204.06125
- [2]Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479–36494, 2022.arxiv.org/abs/2205.11487
- [3]Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.arxiv.org/abs/2112.10752
- [4]OpenAI. Introducing 4o image generation. https://openai.com/index/introducing-4o-image-generation/, 2025. Accessed: 2026-04-30.
- [5]Seedream Team, Yunpeng Chen, Yu Gao, Lixue Gong, Meng Guo, Qiushan Guo, Zhiqiao Guo, Xiaoxia Hou, Weilin Huang, Yixuan Huang, Xiaowen Jian, Huafeng Kuang, Zichao Lai, Fanshi Li, Liang Li, Xiaocheng Lian, Chao Liao, Liyang Liu, Wei Liu, Yanzuo Lu, Zhengxing Luo, Tongtong Ou, Guang Shi, Yichun Shi, Shiqi Sun, Yu Tian, Zhi Tian, Peng Wang, Rui Wang, Xun Wang, Ye Wang, Guofeng Wu, Jie Wu, Wenxu Wu, Yonghui Wu, Xin Xia, Xuefeng Xiao, Shuang Xu, Xin Yan, Ceyuan Yang, Jianchao Yang, Zhonghua Zhai, Chenlin Zhang, Heng Zhang, Qi Zhang, Xinyu Zhang, Yuwei Zhang, Shijia Zhao, Wenliang Zhao, and Wenjia Zhu. Seedream 4.0: Toward next-generation multimodal image generation. arXiv preprint arXiv:2509.20427, 2025.arxiv.org/abs/2509.20427
- [6]Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Sheng-ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, Yuxiang Chen, Zecheng Tang, Zekai Zhang, Zhengyi Wang, An Yang, Bowen Yu, Chen Cheng, Dayiheng Liu, Deqing Li, Hang Zhang, Hao Meng, Hu Wei, Jingyuan Ni, Kai Chen, Kuan Cao, Liang Peng, Lin Qu, Minggang Wu, Peng Wang, Shuting Yu, Tingkun Wen, Wensen Feng, Xiaoxiao Xu, Yi Wang, Yichang Zhang, Yongqiang Zhu, Yujia Wu, Yuxuan Cai, and Zenan Liu. Qwen-image technical report. arXiv preprint arXiv:2508.025?0324, 2025.arxiv.org/abs/2508.02324
- [7]Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20406–20417, 2023.arxiv.org/abs/2303.11897
- [8]Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiaoyao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. In European Conference on Computer Vision, pages 366–384. Springer, 2024.arxiv.org/abs/2404.01291
- [9]Jaemin Cho, Yushi Hu, Roopal Garg, Peter Anderson, Ranjay Krishna, Jason Baldridge, Mohit Bansal, Jordi Pont-Tuset, and Su Wang. Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-to-image generation. In International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=rw9w39vVd4.arxiv.org/abs/2310.18235
- [10]Fanqing Meng, Wenqi Shao, Lixin Luo, Yahong Wang, Yiran Chen, Quanfeng Lu, Yue Yang, Tianshuo Yang, Kaipeng Zhang, Yu Qiao, et al. Phybench: A physical commonsense benchmark for evaluating text-to-image models. arXiv preprint arXiv:2406.11802, 2024.arxiv.org/abs/2406.11802
- [11]Ziwei Huang, Wanggui He, Quanyu Long, Yandi Wang, Haoyuan Li, Zhelun Yu, Fangxun Shu, Weilong Dai, Hao Jiang, Fei Wu, et al. T2i-factualbench: Benchmarking the factuality of text-to-image models with knowledge-intensive concepts. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 27501–27524, 2025.DOI
- [12]Hsin-Ping Huang, Xinyi Wang, Yonatan Bitton, Hagai Taitelbaum, Gaurav Singh Tomar, Ming-Wei Chang, Xuhui Jia, Kelvin C. K. Chan, Hexiang Hu, Yu-Chuan Su, and Ming-Hsuan Yang. Kitten: A knowledge-intensive evaluation of image generation on visual entities, 2024. URL https://arxiv.org/abs/2410.11824.
- [13]Yuwei Niu, Munan Ning, Mengren Zheng, Weiyang Jin, Bin Lin, Peng Jin, Jiaqi Liao, Chaoran Feng, Kunpeng Ning, Bin Zhu, et al. Wise: A world knowledge-informed semantic evaluation for text-to-image generation. arXiv preprint arXiv:2503.07265, 2025.arxiv.org/abs/2503.07265
- [14]Daoan Zhang, Che Jiang, Ruoshi Xu, Biaoxiang Chen, Zijian Jin, Yutian Lu, Jianguo Zhang, Liang Yong, Jiebo Luo, and Shengda Luo. Worldgenbench: A world-knowledge-integrated benchmark for reasoning-driven text-to-image generation. arXiv preprint arXiv:2505.01490, 2025.arxiv.org/abs/2505.01490
- [15]Wenchao Zhang, Jiahe Tian, Runze He, Jizhong Han, Jiao Dai, Miaomiao Feng, Wei Mi, and Xiaodan Zhang. Align beyond prompts: Evaluating world knowledge alignment in text-to-image generation. arXiv preprint arXiv:2505.18730, 2025.arxiv.org/abs/2505.18730
- [16]Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. Advances in Neural Information Processing Systems, 36:78723–78747, 2023.DOI
- [17]Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. Advances in neural information processing systems, 36:36652–36663, 2023.arxiv.org/abs/2305.01569
- [18]Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:15903–15935, 2023.arxiv.org/abs/2304.05977
- [19]Yuhang Ma, Xiaoshi Wu, Keqiang Sun, and Hongsheng Li. Hpsv3: Towards wide-spectrum human preference score. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15086–15095, 2025.arxiv.org/abs/2508.03789
- [20]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021.arxiv.org/abs/2103.00020
- [21]Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 conference on empirical methods in natural language processing, pages 7514–7528, 2021.
- [22]Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36:52132–52152, 2023.arxiv.org/abs/2310.11513
- [23]Kaiyi Huang, Chengqi Duan, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench++: An enhanced and comprehensive benchmark for compositional text-to-image generation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(5):3563–3579, 2025.DOI
- [24]Baiqi Li, Zhiqiu Lin, Deepak Pathak, Jiaxao Li, Yixin Fei, Kewen Wu, Tiffany Ling, Xide Xia, Pengchuan Zhang, Graham Neubig, et al. Genai-bench: Evaluating and improving compositional text-to-visual generation. arXiv preprint arXiv:2406.13743, 2024.arxiv.org/abs/2406.13743
- [25]Yuxuan Luo, Yuhui Yuan, Junwen Chen, Haonan Cai, Ziyi Yue, Yuwei Yang, Fatima Zohra Daha, Ji Li, and Zhouhui Lian. Mmmg: A massive, multidisciplinary, multi-tier generation benchmark for text-to-image reasoning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025.arxiv.org/abs/2506.10963
- [26]Kaiyue Sun, Rongyao Fang, Chengqi Duan, Xian Liu, and Xihui Liu. T2i-reasonbench: Benchmarking reasoning-informed text-to-image generation. arXiv preprint arXiv:2508.17472, 2025.arxiv.org/abs/2508.17472
- [27]Zhichao Yang, Tianjiao Gu, Jianjie Wang, Feiyu Lin, Xiangfei Sheng, Pengfei Chen, and Leida Li. Longt2ibench: A benchmark for evaluating long text-to-image generation with graph-structured annotations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 11820–11828, 2026.
- [28]Jintao Lin, Bowen Dong, Weikang Shi, Chenyang Lei, Suiyun Zhang, Rui Liu, and Xihui Liu. Aegis: Exploring the limit of world knowledge capabilities for unified multimodal models. arXiv preprint arXiv:2601.00561, 2026.arxiv.org/abs/2601.00561
- [29]Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 conference on empirical methods in natural language processing, pages 2511–2522, 2023.arxiv.org/abs/2303.16634
- [30]Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916, 2023.
- [31]Weihao Yu, Zhengyuan Yang, Lijie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. In International Conference on Machine Learning, pages 57730–57754. PMLR, 2024.arxiv.org/abs/2308.02490
- [32]Amita Kamath, Kai-Wei Chang, Ranjay Krishna, Luke Zettlemoyer, Yushi Hu, and Marjan Ghazvininejad. Geneval 2: Addressing benchmark drift in text-to-image evaluation. arXiv preprint arXiv:2512.16853, 2025.arxiv.org/abs/2512.16853
- [33]Hongxiang Li, Yaowei Li, Bin Lin, Yuwei Niu, Yuhang Yang, Xiaoshuang Huang, Jiayin Cai, Xiaolong Jiang, Yao Hu, and Long Chen. Gir-bench: Versatile benchmark for generating images with reasoning. arXiv preprint arXiv:2510.11026, 2025.arxiv.org/abs/2510.11026
- [34]Yibin Wang, Zhimin Li, Yuhang Zang, Jiazi Bu, Yujie Zhou, Yi Xin, Junjun He, Chunyu Wang, Qinglin Lu, Cheng Jin, et al. Unigenbench++: A unified semantic evaluation benchmark for text-to-image generation. arXiv preprint arXiv:2510.18701, 2025.arxiv.org/abs/2510.18701
- [35]Naina Raisinghani. Nano Banana 2: Combining Pro capabilities with lightning-fast speed. Google DeepMind, Google Blog. https://blog.google/innovation-and-ai/technology/ai/nano-banana-2/, February 2026. Published February 26, 2026. Introduces Gemini 3.1 Flash Image (Nano Banana 2). Accessed: 2026-11-06.
- [36]Black Forest Labs. FLUX.2: Frontier visual intelligence. https://bfl.ai/blog/flux-2, November 2025. Published November 25, 2025. Includes FLUX.2 [pro]. Accessed: 2026-11-06.
- [37]Anna Veronika Dorogush. Introducing Recraft V4. Recraft. https://www.recraft.ai/blog/introducing-recraft-v4-design-taste-meets-image-generation, February 2026. Published February 17, 2026. Accessed: 2026-11-06.
- [38]Siyu Cao, Hangting Chen, Peng Chen, Yiji Cheng, Yutao Cui, Xinchi Deng, Ying Dong, Kipper Gong, Tianpeng Gu, Xiusen Gu, et al. HunyuanImage 3.0 technical report. arXiv preprint arXiv:2509.23951, 2025. URL https://arxiv.org/abs/2509.23951.
- [39]Kling AI. Kling IMAGE 3.0 Series Full Rollout. https://kling.ai/release-note/release-notes/rz3idhopum, 2026. Published January 31, 2026. Accessed: 2026-11-06.
- [40]Midjourney. V7 is now the default model! https://updates.midjourney.com/v7-is-now-the-default-model/, June 2025. Published June 17, 2025. Accessed: 2026-11-06.
- [41]ByteDance Seed. Deeper thinking, more accurate generation: Introducing Seedream 5.0 Lite. https://seed.bytedance.com/en/blog/deeper-thinking-more-accurate-generation-introducing-seedream-5-0-lite, February 2026. Published February 13, 2026. Accessed: 2026-11-06.
- [42]Alibaba Cloud. Alibaba unveils Wan2.7 redefining personalized and precision image creation. https://www.alibabacloud.com/en/press-room/alibaba-unveils-wan2-7-redefining-personalized-and-precision-image-creation, April 2026. Published April 1, 2026. Introduces Wan2.7-Image and Wan2.7-Image-Pro. Accessed: 2026-11-06.
- [43]Qwen Team. Qwen-Image-2.0: Professional infographics, exquisite photorealism. https://qwen.ai/blog?id=qwen-image-2.0, February 2026. Published February 10, 2026. Accessed: 2026-11-06.



