WorksComputer Science
Ultimate3D: Teaching Multimodal LLMs 3D Camera-Object Relations by Large-Scale Visual Instruction Data
Abstract
Multimodal Large Language Models (MLLMs) struggle with accurately capturing camera-object relations, especially for object orientation, camera viewpoint, and camera shots. This stems from the fact that existing MLLMs are trained on images with limited diverse camera-object relations and corresponding textual descriptions. To address this, we propose a synthetic generation pipeline to create large-scale 3D visual instruction datasets. Our framework takes 3D assets as input and uses rendering and diffusion-based image generation models to create photorealistic images preserving precise camera-object relations. Additionally, large language models (LLMs) are used to generate text prompts for guiding visual instruction tuning and controlling image generation. We create Ultimate3D, a large-scale 3D-grounded 2D visual instruction dataset. Ultimate3D consists of 240K VQAs with precise camera-object annotations, and corresponding benchmark. MLLMs fine-tuned on our proposed dataset outperform commercial models by a large margin, achieving an average accuracy improvement of 29.7% on camera-object relation recognition tasks. Our code, dataset, and benchmark will contribute to broad MLLM applications.
Introduction
The advent of Multimodal Large Language Models (MLLMs) [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12] have demonstrated remarkable success on tasks including image captioning, embodied robotic manipulation, and visual question answering (VQA). MLLMs bridge visual understanding from vision encoders [13, 14] with powerful verbal capability from large language models (LLMs) [15, 16]. Collectively this provides end-to-end inferencing across image and text modalities. In fact, the recently released Gemini-3.5 [17], GPT-5.5 [18] and Claude-4.7-Opus [19] have extended their performances to be comparable to human intelligence in some of the experimented tasks [20].
However, research works have found needs and weaknesses of MLLMs [21, 22, 23]. Tong et al. [21] point out the deficiency in visual representations learnt by ViT [13]. In particular, two weakness categories (i.e., orientation/direction, viewpoint/perspective) focus on camera-object relations such as distinguishing simple concepts about object orientation (”right”, ”front”, ”back”, etc.), camera viewpoint location (”top”, ”bottom”, etc.), and camera-shot styles (”close-up”, ”long-shot”, etc.). As two examples in Figure 1 show, current SOTA model (i.e., Gemini-3.5) struggle with understanding such camera-object relations. The challenges to improving 3D-aware MLLM abilities on recognizing camera-object relations include: (1) capturing the complexity of the task, and (2) having a dataset containing millions of image-text pairs as is a common requirement for training MLLM on spatial perception tasks [24, 25, 26].

Figure 1. Generating Synthetic Visual Instruction Dataset. Our framework uses open-sourced 3D assets to generate photo-realistic images with precisely controlled camera-object relation. Corresponding text instructions are also generated by Large Language Model (LLM). The generated Ultimate3D dataset (240K) and benchmark (8K) advance baseline MLLM models (Gemma-4, Qwen3-VL, etc.) to outperform commercial MLLMs (GPT-5.5, Gemini-3.5, etc.) on camera-object relation recognition tasks. The left panel illustrates representative MLLM failures, while the right panel provides a direct same-image comparison between base and Ultimate3D-fine-tuned models.
Nevertheless, to the best of our knowledge, none of the existing MLLMs or training datasets
focus on the aforementioned camera-object relations. Prior methods for MLLM training data generation can be divided into the following two groups: (1) Relabeling of existing real images. These works focus on generating textual captions by deep learning models given existing images [27, 28, 29, 30, 31, 32]. Specifically, SpatialRGPT [24] and SpatialVLM [25] address spatial perception and reasoning weakness by providing a comprehensive data annotation pipeline leveraging multiple image segmentation, depth estimation, and camera calibration models to produce dense predictions for real images. But, in practice the accuracy of the predicted labels is not high and the dataset size and distribution is constrained by existing image datasets which predominantly only contain ”front” view objects. (2) Generating synthetic images. Typically text-to-image DMs are utilized for synthetic image generation based on provided text prompts [33, 34, 35]. But, their score-based data curation is not robust to camera-object relations, or generation failures (See Appendix § 11). Alternatively, simulation-based generation pipelines for visual reasoning [36, 37, 38] may provide ground truth camera-object relations, but the generated image quality, complexity, and diversity are still far from realistic. In summary, existing synthetic dataset generation pipelines fail to provide image-text pairs with ground truth camera-object relation, realistic image quality, and diverse image subject and context.
In this work, we propose a synthetic generation pipeline to create unlimited image-text pairs focusing on camera-object relations. With 3D assets (e.g., 3D models with texture, materials, and animation) as input, our framework utilizes a renderer (e.g., Blender) with arbitrary camera-object relation parameters to render a series of 3D ground truth priors (i.e., RGB, depth, segmentation). Those 3D priors are used as comprehensive conditions for stacked diffusion-based generation models (e.g., ControlNet) to generate photorealistic images preserving precise camera-object relations. Simultaneously, LLMs are used to generate versatile text prompts for the controlling of image context and object appearance, and to generate diverse VQA prompts regarding camera-object relations. With our framework, we have created the Ultimate3D dataset as a future benchmark for camera-object relations. The proposed framework serves as a data-generation pipeline that produces photorealistic image-text pairs with precisely controlled camera-object relations from 3D assets. We then use standard fine-tuning protocols to evaluate the effectiveness of Ultimate3D across multiple open-source MLLMs, including LLaVA-, Llama-, Gemma-, and Qwen-based models. Models fine-tuned on Ultimate3D outperform commercial MLLMs by a large margin (i.e., an average 29.7% accuracy improvement) on recognizing camera-object relations.
Our contributions include:
A synthetic generation pipeline for unlimited image-text pair creation of diverse camera-object relations.
A dataset and benchmark for further improving and evaluating camera-object relations in MLLMs.
Extensive evaluation across multiple open-source MLLMs, showing that Ultimate3D fine-tuning consistently improves camera-object relation recognition and outperforms commercial SOTA models.
Related Works
Datasets and Benchmarks for MLLMs
MLLMs use various dataset recipes for visual instruction tuning [1, 2, 3, 4, 5, 6, 7, 8]. Typically, dozens of datasets, focusing on different areas, are combined to enable the comprehensive abilities of MLLMs. Datasets include image captioning [39, 40, 41], visual reasoning [42, 43], general VQA data [3, 44, 45] on knowledge grounding [46, 47], image understanding [48], and OCR [49, 50]. Further, most visual instruction datasets are relabeled or created from other existing image-based datasets (e.g., COCO [39], LAION [51], LVIS [52], Visual Genome [53], ADE20K [54]). As far as we know, none of the datasets focus on camera-object relations. Thus, the missing ground truth camera-object relation data exacerbates the task for deep learning models, such as SpatialRGPT [24] and SpatialVLM [25]. In fact, multiple papers have indicated MLLMs’ poor distinguishing ability on camera-object relations [21, 22]. Our paper fills this gap by providing a generation pipeline, dataset, and benchmark.
In particular for the visual instruction data generation, LLMs, like GPT-4o [55], have been successfully utilized in LLaVA [3, 43] for dialogue content instruction generation, and used in MMVP [21] for answer grading. However, generating high-quality images corresponding to proposed text prompts has not been well developed. In this project, we leverage the diverse text generation capability of GPT-4o for the text prompt generation and we provide an image generation pipeline regarding 3D-awareness.
Simulation-based Image Generation
CLEVR [36] and Kubric [38] use 3D engines to generate synthetic images by rendering 3D primitives at multiple arbitrary camera-object relations. Generally, simulation-based image rendering easily preserve ground truth camera-object relations in arbitrary viewpoint settings. However, the generated images are neither photorealistic nor diverse. Realism is limited by the assets’ level of detail and textures, and diversity is restricted by the small number of assets. These limitations constrain the usage of synthetic images in MLLM training to only object counting or simple spatial reasoning tasks [8]. In this project, we leverage the 3D geometry prior rendered by a 3D renderer to precisely guide the image generation pipeline for preserving ground truth camera-object relation.
Generative Images
Generative models provide advanced image generation by iteratively restoring the original data by denoising from Gaussian noise [56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72]. Open-sourced Stable Diffusion [73], and SDXL [74] enable high-resolution and photo-realistic image generation and editing. While generative models provide versatile context and subject, representative works (e.g., AURORA [75], EditBench [76], Prompt-to-Prompt [77], and MagicBrush [78]) fail to control camera-object relations in the generated content, and quality guarantees rely on human checking. ControlNet [79], built on versatile backbone DMs (e.g., SDXL), provides spatial control in the generated images by using various image priors (e.g., Canny edges, semantic segmentation, relative depth).
In particular, Ma et al. [80] explore using synthetic image generation for 3D pose estimation by conditioning diffusion models on pose angles. However, their approach does not address advancing MLLMs 3D spatial understanding, nor does it consider VQA instruction generation. Moreover, their reliance on 2D Canny priors (i.e., silhouette) brings severe artifacts for images with backward viewpoint and complex depth variations. In contrast, our method incorporates depth priors to enhance geometric accuracy and leverages SDXL’s superior fidelity to improve image realism. Further discussions and user studies are in § 4.6 and § 4.7.
Method
Our framework is described in Figure 2 and Appendix § 20. Given input 3D asset , its category , and arbitrary camera-object relation , our framework generates photorealistic image preserving , and corresponding text QA instruction . Our pipeline consists of 3D visual prior rendering (§ 3.1), diverse image description generation (§ 3.2), controlled image generation (§ 3.3), and 3D-aware text instruction generation (§ 3.4).

Figure 2. Our Framework. Each shade box corresponds to a section in § 3. Given open-sourced 3D assets, our approach leverages a 3D Renderer to generate 3D visual priors () preserving ground truth camera-object relation (). Meanwhile, LLMs take 3D asset category to generate diverse image descriptions () as conditional guidance. Both the 3D visual priors and diverse text prompts are used to generate synthetic images () by multiple ControlNet-based networks. Corresponding text QA instructions () are generated by LLMs given the ground truth camera-object relation. Our Ultimate3D dataset and benchmark (i.e. pairs of and ) contribute to fine-tuning and evaluation of MLLMs.
3D Visual Prior Rendering
Our system renders several 3D visual priors including 2D depth images, Canny-edge images, and segmentation masks from arbitrary viewpoints. These 3D visual priors contain the scene’s geometry and will be used together with LLM-generated image descriptions to collectively control the DM-based image generation in § 3.3.
Camera-Object Relation Definition. As Figure 3 shows, we define camera-object relations using an explicit 3D asset representation and camera-object relative pose.
Figure 3. 3D Visual Prior Rendering. With a general camera model, our method utilizes a 3D Renderer (e.g., Blender) to render multiple 3D visual priors given arbitrary camera-object relations. : camera center; : object center; : camera-object distance.
Each input 3D asset is represented as a textured polygonal mesh , where denotes the mesh vertices, denotes the mesh faces, and contains appearance information such as materials and textures. We associate each asset with an object coordinate frame , whose origin is located at the object center (the red point in Figure 3), whose vertical axis specifies the canonical upright direction, and whose unit vector specifies the canonical object-facing direction. We further define a world coordinate frame and a camera coordinate frame . Let and denote the object-to-world and camera-to-world rigid transformations, respectively. The relative transformation from the object frame to the camera frame is therefore , which corresponds to the standard 6-DoF camera-object relative pose.
In our setting, we constrain the asset to remain upright, fix camera roll, and orient the camera optical axis toward the object center . Under these constraints, the camera-object relative pose can be compactly parameterized by , consisting of three components:
The object orientation angle , which represents the relative azimuth between the asset’s canonical facing direction and the camera-facing direction. Let denote the canonical facing direction transformed into the world coordinate frame, and let denote the world-up direction. For any vector , define its projection onto the horizontal plane as , Let , , where is the camera aperture center. We define
Thus, is the counterclockwise azimuth of the asset-facing direction with respect to the camera-facing direction .
The viewpoint angle , which represents the elevation of the camera with respect to the horizontal plane of the asset. Let denote the vector from the asset center to the camera. We define
The camera-object distance , defined as the Euclidean distance between the camera center and the object center, measured in units of the object’s size,
where is the largest extent of the asset’s axis-aligned bounding box. In practice each asset is rescaled to unit extent before rendering, so equals the camera distance in Blender units. Under our fixed 35mm focal length, corresponds to the object spanning the full image frame, and larger yields a smaller apparent object size.
Thus, should be interpreted as a three-parameter representation of the constrained camera-object relative pose, rather than as a general representation of arbitrary 3D geometry. Compared with [80], this explicit 3D camera-object parameterization allows us to systematically control object orientation, camera elevation, and camera-object distance during dataset generation.
Visual Prior Rendering. Given an arbitrary camera-object relation , the system renders a corresponding set of geometrically aligned 2D visual priors (Figure 3) using a 3D renderer . We use a perspective camera with a default focal length of 35mm, while its extrinsic parameters are determined by . The asset remains upright and the camera is oriented toward its center. The rendered RGB images are further processed to provide Canny edges as additional visual priors.
Because all rendered modalities are produced from the same underlying 3D mesh and camera parameters, they are pixel-aligned and geometrically consistent with the specified camera-object relation . Images rendered under different values of therefore provide controlled variations in object orientation, camera viewpoint, and camera-shot distance while preserving exact 3D geometric supervision.
Diverse Image Description Generation
Our approach uses GPT-4o as a LLM to generate diverse text prompts for image descriptions () for DM-based image generation; an example is in Figure 2. Our framework uses system prompting and few-shot prompting tricks which are broadly utilized for image captioning and QA composing [3, 43].
For the generation of , the LLM is guided to generate diverse descriptions of background context and object details given the object category . The diversity and rationality of the description directly influences the image generation quality (see Appendix § 13). More details are in Appendix § 8.
Controlled Image Generation
We utilize DMs to generate photorealistic images given comprehensive visual prior . A successful generation of shall preserve ground truth camera-object relation information, and be of high fidelity. We first describe a traditional text-to-image DM pipeline, extend it to stacked multiple ControlNets using various images of , and obtain precisely-controlled image generation.
Text-to-Image Generation. We use text-to-image DM (i.e., SDXL-1.0 [74]) as our backbone. The denoising U-Net, parameterized by , conducts iterative denoising diffusion steps of Gaussian noise for a total of time steps. A text-encoder transforms the text prompt into text embedding for guidance injection. The iterative denoising is defined as:
where , and is the latent variable of the generated image. It can be decoded to an RGB image by pretrained image decoder as .
Precise Controlling via Multiple ControlNets. ControlNet [79] guides image generation by injecting additional controlling features derived from various 2D image visual priors. There are dozens of off-the-shelf ControlNets, each pre-trained by a single type of visual prior. In our work, we leverage multiple ControlNets to capture comprehensive visual priors from (depth, Canny edge, etc.) where each one is weighted by coefficients . The denoising process using multiple ControlNets () is defined as:
Similarly, the output generated image and is expected to have ground truth camera-object relations.
Text Instruction Generation
We utilize GPT-4o as a LLM to generate QA pairs on camera-object relation () for visual instruction tuning (more details in Appendix § 9). Specifically, the LLM is prompted to generate QA pairs given the camera-object relation . We subdivide each component of into categorical types for effective referring through textual response by MLLMs:
Object orientation angle is classified into eight object orientation types, each covers a bin of azimuth (i.e. ”right”, ”front right”, ”front”, ”front left”, ”left”, ”back left”, ”back”, ”back right”).
The viewpoint angle is classified into three viewpoint types, each covering a range of elevation angle (i.e. ”horizontal”, ”top”, ”bottom”).
The camera-object distance (§ 3.1) is measured in multiples of the object’s size, with corresponding to the object filling the image frame. The three camera-shot types (i.e. ”close-up”, ”medium-shot”, ”long-shot”) are decided by two cutoff values, 1.25 and 3.0.
Examples of generated multiple-choice style QA pairs as MMVP benchmark [21] are showed in Figure 2. For question generation, we prompt LLMs to generate template questions for each prediction task of object orientation, viewpoint type, and camera-shot type. We list all possible options, including the ground truth , in random order. MLLMs are expected to choose the option according to .
Experiments
Visual Instruction Generation
Data Source. Our framework uses open-sourced 3D assets datasets from ObjaverseXL [81, 82] and ShapeNet [83]. Partial 3D assets are labeled following the synset list from ImageNet-1K [84]. We leverage this subset and define the asset category by its synset label. In particular, we manually select 100 synsets with explicit ”facing” orientation from the original list of ImageNet. In total, there are 1196 assets used for synthetic visual instruction generation.
ControlNets. We utilize SDXL-1.0 [74] as the backbone of ControlNets. For selection of multiple visual priors, we found using both Canny edges and relative depth provides the best mix of camera-object relation preservation and image quality. The diffusion step , and relative weights for depth and Canny edge ControlNets are 0.5 and 0.8, respectively. Several seconds are needed to generate an image by a single H100 GPU. Ablations and user studies are shown in § 4.6 and § 4.7.
Ultimate3D Dataset and Benchmark.
Our framework creates the Ultimate3D, a large-scale 3D-grounded 2D visual instruction dataset whose samples are generated from explicit 3D assets and controlled camera geometry, providing precise ground-truth camera-object relation annotations. Ultimate3D consists of 85K synthetic images and 18K cropped images of a single human per photograph and corresponding human orientation labels from MEBOW [85]. For the synthetic images, we use 1180 3D assets covering 100 categories. For each 3D asset, there are 72 possible camera-object relations (8 orientations, 3 viewpoints, and 3 camera shot types) and we render a corresponding synthetic image for each relation. Then, each synthetic image is described by up to 3 textual QAs corresponding to object orientation, camera viewpoint, and camera-shot type, yielding a total of 240K corresponding QAs. Examples of synthetic images are in Appendix § 16. The images from MEBOW are useful because otherwise there is a strong lack of 3D assets of human which we find crucial to fine-tuning. The MEBOW images only have one QA corresponding to human orientation.
The Ultimate3D benchmark (Figure 4) has two sets: (1) Synthetic Set. It contains 1200 synthetic images and 3600 QAs generated by our framework. Each image and QA has been manually reviewed to ensure correctness. This set covers all 100 categories with explicit facing orientation. The 3D assets utilized for benchmark generation do not overlap with those used for dataset generation. (2) Real Set. It contains two portions. The first portion is 2443 real images collected from Pascal3D+ [86] covering 10 categories (e.g. aeroplane, bicycle, etc.). Each image corresponds to a QA based on provided labels of object orientation or camera viewpoint. The second portion is 800 real images of humans collected from the MEBOW dataset [85]. Each image corresponds to a QA on human orientation based on provided labels. In total, there are 3243 VQAs in the Real set. We resample the real images to keep a uniform distribution for each orientation and viewpoint type.

Figure 4. Qualitative Results. We show responses by GPT-5.5, Gemini-3.5-Flash with high thinking, Gemma-4-31B-it, and finetuned Gemma-4-31B-it on the Ultimate3D benchmark. Each model is asked questions (in gray boxes) regarding object orientation, camera viewpoint, and camera-shot type, together with the input images on the left. The model responses illustrate that the Gemma-4-31B-it model finetuned on Ultimate3D can precisely recognize camera-object relations on synthetic images. GPT-5.5 and Gemini-3.5-Flash with high thinking sometimes provide correct answers, but also exhibit inconsistent predictions across relation types. The original Gemma-4-31B-it frequently fails to produce the correct camera-object relation labels, while the finetuned model provides substantially more accurate and task-relevant answers.
Since no camera-shot information is provided in the real image set, the QAs in the real set exclude camera-shot type questions.
Both the Ultimate3D dataset and the benchmark have no camera-object relation bias, which is very uncommon in the real image datasets (e.g. most images capture objects facing straight to the camera [86]). In contrast, in our dataset the number of images in each orientation, viewpoint, and camera-shot type are evenly distributed.
Advancing 3D-Aware MLLMs
We explore how much our synthetic Ultimate3D dataset can improve the MLLMs recognition capability on camera-object relation. The evaluation results on both synthetic and real image VQA benchmarks indicate the fine-tuned baseline models are significantly improved from random guess and outperform commercial SOTAs.
Training Details. We fine-tune and evaluate several recent open-source MLLM families, including LLaVA-1.5/1.6 [4], Llama-3.2-Vision-11B [9], Gemma-4 models [10], and Qwen-based models [11, 12]. All models are fine-tuned for one epoch on the Ultimate3D training set. To evaluate the effect of training on Ultimate3D, we compare each base model with its corresponding fine-tuned version. We adopt two fine-tuning protocols depending on the model family. For LLaVA and Llama-3.2-Vision, we follow their official full fine-tuning recipes [4, 9], updating the MLP vision-language connector and the LLM decoder while freezing the vision encoder. For LLaVA, we use BF16 precision, a learning rate of , cosine scheduling with a warm-up ratio of 0.03, zero weight decay, and gradient checkpointing. For Llama-3.2-Vision-11B, we use BF16 precision, fused AdamW, a learning rate of , a per-device batch size of 2 with gradient accumulation of 4, and gradient clipping at 1.0. For Gemma-4 and Qwen-based models, we use parameter-efficient fine-tuning primarily based on LoRA. For the Qwen3.5 and Qwen3-VL series, we use Unsloth-based parameter-efficient fine-tuning with a hybrid trainable-parameter policy: the vision encoder is kept frozen, the language-model attention and MLP projections are adapted with LoRA (rank 16, =16, dropout 0, no bias terms, no RSLoRA), and the vision–language connector (merger) is optimized as full weights.
(merger) is optimized as full weights. For Gemma-4 models, we use plain LoRA with rank 16, = 32, and dropout 0.05, applied to compatible linear layers in the language model and vision embedding projection. Gemma-4 models are trained in BF16 using fused AdamW, an effective batch size of 4, and a constant learning rate of without warmup or weight decay. For Gemma-4-31B, we resumed from the best intermediate checkpoint with a learning rate of , cosine decay, and weight decay of 0.01 due to late-stage instability. Training takes approximately 12 hours on 4 H100 GPUs for LLaVA and Llama-3.2-Vision, up to 2.5 days on a single H100 GPU for Gemma-4, and up to 8 days on a single RTX 6000 GPU for Qwen-based models.
Evaluation Settings. We evaluate the baseline MLLMs and their fine-tuned versions on Ultimate3D benchmark and MMVP benchmark [21]. MMVP benchmark is an intentionally selected real image dataset which is claimed to be difficult distinguished by MLLMs. We manually label 89 VQAs regarding camera-object relations. During evaluation, MLLMs’ textual responses are evaluated by GPT-4o, which is prompted to judge if the responses correspond to the ground truth answers. We calculate the accuracy rate (%) of the MLLMs responses as the evaluation metric. We also include commercial SOTA models (e.g. GPT-4o and Claude-3.5-Sonnet) in the evaluation.
Quantitative Results. The quantitative results of response accuracy (%) are shown in Table 1. We assess the effect of training on Ultimate3D through base-to-fine-tuned comparisons within each model family. Overall, we observe a consistent trend: off-the-shelf MLLMs struggle with camera-object relation recognition, whereas task-specific fine-tuning on Ultimate3D substantially improves performance across multiple model families. This difficulty appears even for strong general-purpose vision-language models, including LLaVA, Llama-3.2-Vision, Gemma-4, Qwen-based models, and closed commercial MLLMs. Several baseline models tend to collapse to a small set of frequent answers, such as predicting “front” orientation or “top” viewpoint regardless of the actual image content. As a result, their accuracies are often close to random-guess levels, e.g., 33.3% for three-way viewpoint recognition and 12.5% for eight-way orientation recognition. This weakness may be related to biases in commonly used visual
Table 1: Quantitative Comparisons. We evaluate open-source and closed-source MLLMs on the Ultimate3D and MMVP benchmarks. LLaVA and Llama models are fine-tuned using their original full fine-tuning recipes, while Gemma-4 and Qwen models are fine-tuned with LoRA-based parameter-efficient fine-tuning (for Qwen, the vision-language merger is also fully trained). Results report MLLM response accuracy (%) for object orientation, camera viewpoint, and camera-shot recognition. Superscripts indicate the fine-tuning setup: F full fine-tuning and L LoRA fine-tuning.
| MLLMs | Orientation (% ↑) | Viewpoint (% ↑) | Camera-Shots (% ↑) | |||||||
| Ultimate3D | MMVP | Ultimate3D | MMVP | Ultimate3D | MMVP | |||||
| Both | Syn | Real | Both | Syn | Real | Syn | ||||
| LLaVA-1.5-7B | 18.3 | 11.1 | 22.5 | 25.0 | 31.8 | 38.4 | 25.3 | 29.2 | 46.0 | 66.7 |
| LLaVA-1.6-7B | 18.2 | 14.7 | 20.3 | 25.0 | 36.8 | 43.9 | 29.8 | 33.3 | 46.6 | 71.4 |
| LLaVA-1.5-13B | 17.9 | 18.7 | 17.4 | 22.7 | 38.9 | 35.0 | 42.8 | 41.7 | 44.6 | 61.0 |
| LLaVA-1.6-13B | 16.1 | 16.4 | 16.0 | 15.9 | 30.7 | 30.3 | 31.1 | 33.3 | 42.3 | 61.9 |
| Llama-3.2-V-11B | 5.6 | 5.3 | 5.7 | 6.8 | 19.8 | 26.2 | 13.6 | 38.1 | 29.6 | 25.0 |
| Finetuned LLaVA-1.5-7B | 68.8 | 85.6 | 58.8 | 54.5 | 70.6 | 81.3 | 60.0 | 66.7 | 94.0 | 66.7 |
| Finetuned LLaVA-1.6-7B | 71.5 | 86.9 | 62.4 | 61.4 | 71.3 | 83.3 | 59.5 | 66.7 | 94.1 | 50.0 |
| Finetuned LLaVA-1.5-13B | 70.0 | 85.6 | 60.8 | 50.0 | 69.7 | 81.9 | 57.7 | 66.7 | 94.2 | 76.2 |
| Finetuned LLaVA-1.6-13B | 72.4 | 88.1 | 63.1 | 65.9 | 72.3 | 83.0 | 61.8 | 75.0 | 94.8 | 71.4 |
| Finetuned Llama-3.2-V-11B | 74.2 | 87.3 | 66.4 | 50.0 | 69.1 | 79.6 | 58.7 | 58.3 | 93.1 | 76.2 |
| Qwen3.5-2B | 28.0 | 36.1 | 20.2 | 34.1 | 40.6 | 37.1 | 44.1 | 50.0 | 45.2 | 42.9 |
| Qwen3.5-4B | 37.0 | 42.8 | 31.3 | 45.5 | 45.7 | 64.3 | 27.3 | 50.0 | 45.5 | 47.6 |
| Qwen3.5-9B | 39.6 | 48.3 | 31.1 | 40.9 | 47.1 | 63.7 | 30.8 | 50.0 | 37.7 | 28.6 |
| Qwen3-VL-4B | 21.9 | 28.6 | 15.3 | 45.5 | 45.7 | 52.1 | 39.4 | 33.3 | 43.4 | 47.6 |
| Qwen3-VL-8B | 34.3 | 41.9 | 26.9 | 52.3 | 47.1 | 59.7 | 34.6 | 45.8 | 44.9 | 38.1 |
| Qwen3-VL-32B | 41.1 | 46.7 | 35.6 | 56.8 | 52.5 | 66.4 | 38.8 | 50.0 | 44.3 | 57.1 |
| Finetuned Qwen3.5-2B | 68.0 | 80.4 | 60.2 | 61.4 | 66.8 | 76.5 | 57.1 | 62.5 | 91.7 | 70.7 |
| Finetuned Qwen3.5-4B | 71.3 | 79.0 | 63.5 | 60.9 | 71.0 | 79.0 | 63.0 | 66.7 | 92.1 | 72.8 |
| Finetuned Qwen3.5-9B | 73.3 | 82.5 | 64.0 | 63.6 | 71.8 | 73.5 | 70.2 | 62.8 | 91.9 | 71.2 |
| Finetuned Qwen3-VL-4B | 67.5 | 83.8 | 56.2 | 68.2 | 72.8 | 81.5 | 59.5 | 75.1 | 91.5 | 73.3 |
| Finetuned Qwen3-VL-8B | 69.5 | 80.8 | 62.5 | 62.7 | 71.2 | 76.2 | 67.1 | 70.8 | 89.7 | 77.0 |
| Finetuned Qwen3-VL-32B | 73.1 | 83.5 | 65.5 | 69.5 | 73.4 | 82.3 | 65.1 | 73.3 | 92.2 | 78.7 |
| Gemma-4-E2B-it | 25.6 | 28.2 | 24.0 | 40.9 | 39.5 | 48.2 | 30.9 | 37.5 | 37.7 | 47.6 |
| Gemma-4-E4B-it | 29.8 | 35.2 | 26.6 | 38.6 | 44.0 | 58.0 | 30.1 | 58.3 | 39.2 | 28.6 |
| Gemma-4-26B-A4B-it | 47.3 | 54.6 | 43.0 | 56.8 | 52.6 | 64.2 | 41.1 | 66.7 | 36.9 | 47.6 |
| Gemma-4-31B-it | 44.9 | 55.2 | 38.9 | 52.3 | 52.8 | 63.7 | 42.1 | 54.2 | 40.1 | 52.4 |
| Finetuned Gemma-4-E2B-it | 67.3 | 80.0 | 59.8 | 61.4 | 66.7 | 76.8 | 56.7 | 62.5 | 91.8 | 71.4 |
| Finetuned Gemma-4-E4B-it | 65.7 | 78.6 | 58.2 | 59.1 | 69.4 | 76.2 | 62.8 | 58.3 | 89.8 | 76.2 |
| Finetuned Gemma-4-26B-A4B-it | 71.9 | 83.2 | 65.3 | 61.4 | 67.7 | 76.8 | 58.6 | 58.3 | 92.4 | 71.4 |
| Finetuned Gemma-4-31B-it | 73.9 | 86.8 | 66.3 | 45.5 | 69.8 | 80.1 | 59.7 | 62.5 | 92.2 | 76.2 |
| GPT-4o | 43.1 | 51.0 | 38.5 | 40.9 | 54.1 | 63.5 | 44.9 | 70.8 | 41.7 | 42.9 |
| GPT-4o-mini | 43.5 | 52.3 | 38.3 | 38.6 | 54.5 | 63.9 | 45.1 | 66.7 | 41.5 | 42.9 |
| Claude-3-Sonnet | 43.1 | 52.6 | 37.5 | 36.4 | 54.3 | 64.2 | 44.5 | 66.7 | 41.7 | 47.6 |
| Claude-3.5-Sonnet | 40.3 | 47.7 | 35.8 | 54.5 | 55.6 | 65.9 | 45.3 | 66.7 | 41.8 | 42.9 |
| Gemini-3.5-Flash (min. thinking) | 56.8 | 66.9 | 50.8 | 61.4 | 61.4 | 68.4 | 54.4 | 83.3 | 37.1 | 52.4 |
| Gemini-3.5-Flash (high thinking) | 65.3 | 73.8 | 60.3 | 56.8 | 61.6 | 67.0 | 56.3 | 79.2 | 33.8 | 42.9 |
| Claude-Sonnet-4.6 | 36.4 | 44.1 | 28.9 | 50.0 | 56.4 | 67.1 | 45.8 | 66.7 | 40.9 | 47.6 |
| GPT-5.5 | 54.1 | 62.5 | 45.9 | 50.5 | 59.2 | 69.7 | 48.9 | 58.3 | 37.3 | 52.4 |
| Claude-Opus-4.7 | 46.0 | 52.0 | 40.4 | 45.5 | 57.2 | 67.1 | 47.5 | 62.5 | 44.3 | 52.4 |
Table 1.
instruction datasets, where most images are captured from canonical front-facing directions and from common side or above viewpoints. The poor off-the-shelf performance across both older and more recent MLLM families suggests that camera-object relation understanding remains underrepresented in existing MLLM training data.
Fine-tuning on our synthetic Ultimate3D dataset substantially improves camera-object relation recognition across all evaluated open-source model families. On the Ultimate3D benchmark, fine-tuned LLaVA models significantly outperform their corresponding base models across object orientation, camera viewpoint, and camera-shot recognition. For example, fine-tuned LLaVA-1.6-13B improves orientation accuracy from 16.1% to 72.4%, viewpoint accuracy from 30.7% to 72.3%, and camera-shot accuracy from 42.3% to 94.8%. The smaller LLaVA-1.6-7B model achieves similarly competitive performance, reaching 71.5%, 71.3%, and 94.1% on these three tasks, respectively. This suggests that the ability to distinguish camera-object relations does not necessarily require a very large backbone, but instead benefits strongly from task-specific supervision.
The same conclusion holds for Llama-3.2-Vision, Gemma-4, and Qwen-based models. Llama-3.2-Vision-11B obtains the largest average improvement among the models in Table 1, with an average gain of 60.46 percentage points across the three camera-object relation tasks. The original released model often refuses to answer some recognition questions, e.g., responding that it cannot provide information about the individual in the image, while the fine-tuned model produces valid relation predictions and achieves the best orientation accuracy in Table 1. These results indicate that Ultimate3D supervision can correct both answer-format failures and visual relation recognition failures.
Recent Gemma-4 and Qwen-based models further confirm that the benefit of Ultimate3D supervision is not specific to the LLaVA architecture. Before task-specific fine-tuning, Gemma-4 models achieve only 36.0–51.1% overall accuracy when averaging all reported relation-recognition columns. After LoRA fine-tuning, they reach 69.4–71.3% overall accuracy, corresponding to an average improvement of 26.3 percentage points. The improvements are consistent across all three relation types, with average gains of 20.0–37.5 percentage points on object-direction recognition, 9.2–26.7 percentage points on viewpoint recognition, and 38.0–49.1 percentage points on camera-shot recognition.
Qwen-based models show a similar pattern. Their base versions achieve 37.3–48.9% overall accuracy, while the fine-tuned versions reach 69.5–75.7% overall accuracy, improving by 30.5 percentage points on average. In particular, fine-tuned Qwen3-VL-32B achieves the strongest overall performance among the evaluated open models in Table 1, reaching 75.7% average accuracy across all reported columns. It also obtains strong performance on both synthetic and real subsets, suggesting that the learned camera-object relation knowledge is not restricted to rendered synthetic images.
Among closed commercial MLLMs, newer models generally perform better than earlier GPT-4o- and Claude-3-era models on some orientation and viewpoint columns, but the improvement is not uniform across tasks. For example, Gemini-3.5-Flash with the high thinking setting achieves stronger orientation accuracy than its minimal-thinking counterpart, improving from 56.8% to 65.3% on the Ultimate3D orientation column. However, increasing the thinking budget does not consistently improve all relation types: its camera-shot accuracy decreases from 37.1% to 33.8%. This suggests that additional reasoning budget can help in some spatial-relation cases, but it does not resolve the lack of grounded camera-object relation understanding.
Compared with closed commercial MLLMs, our fine-tuned open models achieve especially strong performance on the Ultimate3D relation-recognition tasks. Although recent commercial models improve over earlier commercial baselines on some orientation and viewpoint columns, they remain substantially weaker on several Ultimate3D tasks, especially camera-shot recognition, where fine-tuned open models consistently reach around 90% accuracy or higher. Overall, these results demonstrate that Ultimate3D provides effective supervision for camera-object relation understanding, improves both synthetic and real-image evaluation performance, and generalizes across multiple MLLM families, including LLaVA, Llama, Gemma, and Qwen.
Qualitative Results.
We show qualitative comparisons in Figure 4. Commercial SOTA models (e.g. Gemini-3.5-Flash) and open-source models (e.g. Gemma-4-31B-it) struggle with precise recognition of all three types of camera-object relations. For the two commercial models, their object-orientation predictions are often unreliable despite producing plausible natural-language explanations. We also observe systematic biases in viewpoint and camera-shot recognition: both commercial models tend to underestimate viewpoint angle, e.g., mistaking top views as horizontal, and overestimate camera-shot distance, e.g., mistaking medium-shot images as long-shot. The original Gemma-4-31B-it model also shows strong response biases across all types of questions. Its orientation predictions are biased mostly toward “Left” and “Right”, its viewpoint predictions are dominated by “Horizontal”, and its camera-shot predictions are biased toward “Long-shot”. In comparison, the Gemma-4-31B-it model fine-tuned on our Ultimate3D dataset provides more plausible predictions across all three tasks on both the synthetic and real subsets of our benchmark. Additional comparisons are provided in Supplemental Sec. H.
We hypothesize that these behavioral biases arise from dataset bias and the lack of camera-object visual instruction data in current MLLM training pipelines. Without modifying the model architecture or training scheme, our fine-tuned Gemma-4-31B-it model substantially improves camera-object relation recognition and outperforms strong commercial models on several Ultimate3D tasks, despite relying only on task-specific supervision. This indicates that the lack of targeted camera-object relation data is a major factor behind current MLLMs’ weakness on camera-object relation recognition.
Limitations. The numerical prediction of camera-object relation is more challenging for MLLMs compared to the categorical prediction we report. Preliminary tests show MLLMs can handle numerical prediction of camera-object relation for a single category, but struggle with a hundred categories as in our dataset. Since our main contribution is the discovery of the dataset bottleneck and the generation pipeline, we leave addressing this limitation as future work.
Generalization of Ultimate3D to General VQA
To explore the generalization of 3D camera-object relation capturing with general VQA tasks, we fine-tune LLaVA-1.6-13B models by a half-half mixture of Ultimate3D (240K VQAs) and randomly sampled 240K VQAs from LLaVA-Instruct-665K [4]. Table 2 presents the dataset mixture finetuning performance on VQAv2 [48], our Ultimate3D, and MMVP benchmarks. The results demonstrate that incorporating Ultimate3D significantly enhances camera-object relation reasoning while maintaining general VQA performance.
Table 2. Generalization. We finetune LLaVA-1.6-13B model using a mixture of both general VQA and camera-object relation VQA datasets. Without degradation on general VQA tasks, fine-tuned model improves its ability on camera-object relation recognition and beats SOTA (i.e. GPT-4o) by 28.42% accuracy on Ultimate3D benchmark. * denotes the average accuracy (%) across all 3 types of camera-object relation VQAs in Table 1.
Ablations on Image Generation Pipeline
In Figure 5, we ablate our image generation pipeline and illustrate better quality than [80] by: (1) removing the visual priors; and (2) using alternative image generation backbones; (3) changing the group of visual priors given to the image generator. Compared to text-only SDXL and [80], we find that multiple visual priors may significantly improve the robustness of image generation under complex camera-object relation (especially for viewpoints of the object’s back-side, and for 3D assets with low level-of-detail). In particular, the depth and Canny edge prior are crucial. However, we find that adding more priors (e.g., RGB, semantic masks, normal maps) does not explicitly improve quality. Moreover, our SDXL backbone

Figure 5. Ablations on Image Generation Pipeline. Our framework shows superior performance by using multi-ControlNet with SDXL backbones, and introducing both Canny edges and depth maps as visual priors. It consistently delivers better robustness and quality compared to text-only SDXL and [80]. When these visual priors are removed (top-right: removing all visual priors; bottom-right: removing depth prior), the generator often fails to produce good results and brings lower generation success rate (see § 4.7, [80] only achieves 55.11% success rate v.s. ours 93.07%) outperforms other alternatives (i.e. SD-V1.5 [73]). Discussions in Appendices § 12 and § 13.
User Study on Dataset Quality
We investigate using text-image aligning metric and pose-estimation model prediction to evaluate quality of our Ultimate3D dataset but find those metrics to be deficient (Appendix § 11). Instead, we perform a comprehensive user study as a quality check of Ultimate3D dataset. As Appendix § 14 shows, the user will see two images side-by-side (Figure G4): an RGB image rendered by Blender (representing ), and a synthetic image generated by our method using . Preserving the RGB image’s 3D geometry and structure in the synthetic image is regarded as success. Our study includes 225 randomly sampled image pairs where each image is independently reviewed by 3 users.

Figure G4. User Study UI Page. Users will see a pair of side-by-side images. The left hand side is an RGB image generated by Blender, and the right hand side is the synthetic image generated by DMs using that RGB image as guidance prior. A vivid preserving of 3D geometry and structure in the synthetic image is a success. Users are required to provide a binary answer regarding the success of the image generation.
Overall success rate is 93.07% (agreement rate among 3 users is 94.4%). In comparison, we perform another similar-objective user study on [80]. The overall success of this alternative method was only 55.11% (we explain the cause of this reduced performance in § 4.6).
Conclusion
In this work, we discover the dataset bottleneck of MLLMs for distinguishing camera-object relations. Given only 3D assets, our framework is able to generate unlimited image-text pairs of diverse camera-object relations with no distribution bias. We also provided Ultimate3D dataset and benchmark MLLM community. Moreover, fine-tuned open-sourced MLLMs using our dataset outperform SOTA commercial models by a significant margin. We plan to extend our framework for multi-object image generation with additional viewpoints, and improve numerical prediction capability on camera-object relation.
Acknowledgments
This work was performed during Liu He’s internship at Amazon, and later extended by Ruiqi Xu and Baris Tezcan at Purdue University. Ruiqi Xu fine-tuned Qwen3 series and evaluated GPT5.5 & Claude models, and Baris Tezcan fine-tuned Gemma-4 series and evaluated Gemini models. We thank Jingjing Zheng, Tony Qi, Nan Qiao, Ke Zhang, and Yuyin Sun for their insightful discussions and valuable contributions to the formulation of the ideas in this work.
Appendix Overview
Below is a summary of the contents in each section of this supplemental material:
Sec. A: Details of text prompt generation as image description utilized for controlled image generation.
Sec. B: Details of VQA prompt generation given the ground truth camera-object relations.
Sec. C: Details of LLM-based grading for MLLM response evaluation.
Sec. D: Preliminary studies on data curating for camera-object relation VQA dataset by text-image aligning metrics, and by pose estimation models.
Sec. E: Ablations on selections of DM backbones.
Sec. F: Factors influence generation quality of our image generation pipelines.
Sec. G: Details of user study on image quality evaluation.
Sec. H: More qualitative comparisons between finetuned LLaVA and Gemma models to commerical SOTA models.
Sec. I: 48 image examples illustrate the diversity of Ultimate3D dataset and benchmarks on object categories, camera-object relations, and background context.
Sec. J: Generalization to real images: evaluation of base and Ultimate3D-fine-tuned models on the independently collected ImageNet3D benchmark.
Sec. K: Failure analysis of fine-tuned models: residual weak relations, error structure, boundary effects, and annotated image conditions.
Sec. L: Additional discussions and insights on using synthetic dataset generated by our framework for MLLM finetuning.
Sec. M: Algorithm of our 3D visual instruction dataset generation pipeline.
Image Description Generation
We provide system prompts given to GPT-4oto generate context description for image generation. Corresponding section in main paper is § 3.2. In the prompt, we provide one-shot example for better robustness of the text generation.
Specifically, generated text prompt will be merged with default positive prompt like ”detailed, 4K, 35mm photograph, professional”, and customized camera-object prompt ”the image shows a [β] view of a [c]”, in order to enhance image generation quality.
SYSTEM_PROMPT_FOR_IMAGE_GEN = "You are an expert in
generating concise and diverse descriptions of an
object to guide image generation model like DAll-E3.
You should use your common sense to generate 1
sentence description of the given object.
Additionally, you should also generate 1 sentence of
a common real-world scene given the object. Both
sentences of descriptions should not include more
than one of the given object to avoid ambiguity.
Example Input: Please generate the visual prompt of
chicken.
Example Output: A white and brown chicken standing in
a cage made of metal bars. The chicken has a long,
curved beak and its feathers are fluffy and white. "
Text Instruction Generation
We provide the system prompt to generate QA pairs given camera-object relation as below. Corresponding section in main paper is § 3.4. In the prompt, we provide few-shot example for better instructions for diversity of the generated text.
SYSTEM_PROMPT_FOR_VQA_GEN = "You are an AI visual
assistant, and you are seeing a single image. What
you see are provided with several sentences,
describing the same image you are looking at. Answer
all questions as you are seeing the image.
Design a conversation between you and a person asking
about this photo. The answers should be in a tone
that a visual AI assistant is seeing the image and
answering the question.
Ask diverse questions and give corresponding answers.
Include questions asking about the visual content of
the image, including the object orientation and its
corresponding azimuth degree, the camera viewpoint
of the image and corresponding camera elevation
angle, the camera shot type of the image and the
distance from camera to the target object. Only
include questions that have definite answers:
(1) one can see the content and its orientation degree
in the image that the question asks about and can
answer confidently;
(2) one can recognize camera viewpoint and camera shot
of the image that the question asks about and can
answer confidently;
The question can be multiple choice questions given
the choice, or open-ended questions. For multiple
choice, the option expression can be diverse but
with the same meaning. For example, "front" also
means "directly towards the camera"; "back" also
means ’away from the camera’, etc. Use the common
sense to make options diverse if possible.
Provide at most 5 pairs of question and answer pairs.
Prior the confident questions. Do not ask any
question that cannot be answered confidently.
You should ask questions from multiple QA templates as
the below examples show.
Example Input Description:
The image shows a front view of police van. A blue and
white police van with the word "POLICE" emblazoned
on the side is parked on a city street. Nearby, a
couple of officers are discussing a recent incident
while pedestrians walk by, some glancing curiously
at the vehicle.
The police van is with an azimuth of 181.518 degree, facing Front direction. The elevation angle of the camera to the police van is 81.41 degree, the camera viewpoint is Horizontal view. The relative distance from the camera to the police van is 1.132 meters, the camera shot type is Close-up.
Example Output Question and Answer Pairs:
Question:
From the camera’s perspective, is the police van in the picture facing straight or oriented at an angle?
Options: (a) Directly towards the camera (b) At an angle
===
Answer:
(a) Directly towards the camera
===
Question:
Is the police van in the picture facing the camera or away from the camera? Options: (a) Away from the camera (b) Facing the camera
===
Answer:
(b) Facing the camera
===
Question:
Which direction is the police van facing in the image?
Options: (1) Back (2) Front
===
Answer:
(2) Front
===
Question:
Is the police van facing back or front from the camera’s perspective? Options: (a) Back (b) Front
===
Answer:
(b) Front
===
Question:
Is the photo taken directly above the police van or from the side? Options: (a) Taken directly (b) From the side
===
Answer:
(b) From the side
===
Question:
Is the photo taken far away the police van or taken closely?
===
Answer:
The relative distance from the camera to the police van is 1.132 meters, thus it is with a close-up shot. This indicates the photo is taken closely.
===
Question:
What is the elevation viewpoint of the image? Options: (1) Top (2) Horizontal (3) Bottom
===
Answer:
(2) Horizontal
===
Question:
Which camera shots is the image? Options: (1) Close-up (2) Medium-shot (3) Long-shot
===
Answer:
(1) Close-up"
LLM-based Response Grading
We provide the system prompt for evaluation of the MLLM response to our camera-object relation multiple choice questions.
SYSTEM_PROMPT_FOR_GRADING_MLLM_RESPONSE =’You are a helpful and precise assistant for checking the quality of the answer. You should review all listed choices and compared to the Answer content, and judge whether the Answer is correct (yes), or not (no). You should only focus on the major content of the answer, not the detailed number or symbol. Please answer in only yes or no’
Metric Limitations for Evaluating Camera-Object Relation
In Figure D1, we show an example of using ImageReward [87] for the dataset curation by evaluating the alignment between generated image and text prompts. Preliminary test indicates that text-image aligning metric is not sensitive to camera-object relation text prompts. Thus we may not rely on those metrics for data curation purpose.

Figure D1. Limitations of Text-Image Aligning Metrics. We evaluate the ImageReward [87] (IR) between the text prompt regarding object orientation with above 4 synthetic images. The image matches the correct camera-object relation text prompt (”front”) may reach the highest score. However, image with the highest matching score (it faces to ”left” direction) is not the correct one with expected orientation. This failure is prevailing across all categories.
Alternatively, in Figure D2, enlighten by [80], we train a transformer-based pose estimation model on a 10-fold manner for generated images on each single category. The images with less prediction errors may indicate successful generation. However, the pose estimation model is not robust to complex objects and diverse camera-object relations in our dataset. Using pose estimation model prediction error is not robust for data curation purpose.

Figure D2. Failures of Pose Estimation Model. Enlighten by [80], we train a transformer-based pose estimation model on a 10-fold manner for generated images on each single category. The images with less prediction errors may indicate successful generation. However, the pose estimation model is not robust to complex objects and diverse camera-object relations in our dataset.
Ablation of DM Backbones
In Table E1, we perform quantitative comparisons on general image visual quality between different DM backbones: SD V1.5 and SDXL. We randomly sample 1,000 generated images (), each image corresponds to: (1) the RGB visual prior rendered using Blender (one element of ); (2) the text prompts () provided to the diffusion model
| Ablations | CLIP-T ↑ | CLIP-I ↑ | DINO ↑ | IR ↑ | FID ↓ |
| Ours (SD-V1.5) | 29.80 | 83.56 | 69.36 | -0.29 | 289.99 |
| Ours (SDXL) | 34.40 | 87.44 | 70.79 | 0.23 | 256.94 |
Table E1. Ablations on DM Backbones. SDXL provides better fidelity (CLIP-I, DINO), realism (FID), and text-image alignment (CLIP-T, IR) than other alternatives.
for image generation. We evaluate the generated images on the following aspects:
Prompt following. CLIP-T (CLIP-Text Score [88]) and IR (Image Reward [87]) metrics are employed to assess the alignment between the generated images and the input prompts. The results indicate that images produced by the SDXL backbone exhibit superior prompt alignment.
Fidelity. CLIP-I (CLIP-Image Score [88]) and DINO Score [14] are used to measure the similarity between the visual prior images and the synthesized outputs. Findings demonstrate that SDXL more effectively preserves fidelity to the visual prior.
Realism. FID [89] is used to compare the distributions of generated images with a subset of LAION-aesthetic [51] images, revealing that SDXL significantly surpasses SD V1.5 in terms of image quality and realism.
Factors influencing Image Generation
Several factors may lead to unreasonable artifacts in our image generation pipeline. In Figure F3, the upper-left example shows that bottom-view camera-object relations can introduce physical anomalies, such as the tricycle appearing with its front lifted unnaturally. The upper-right example highlights that low-quality 3D assets can produce fragmented depth maps or Canny edge visual priors, resulting in unrealistic generated images. The lower-left example shows that small subjects viewed from a long-shot perspective may lead to duplicate subjects in the generated image. The lower-right example illustrates how complex 3D asset structures can cause texture blending issues, particularly where intricate geometry is present.

Figure F3. Factors influencing Image Generation. The upper-left example shows that bottom-view camera-object relations can introduce physical anomalies, such as the tricycle appearing with its front lifted unnaturally. The upper-right example highlights that low-quality 3D assets can produce fragmented depth maps or Canny edge visual priors, resulting in unrealistic generated images. The lower-left example shows that small subjects viewed from a long-shot perspective may lead to duplicate subjects in the generated image. The lower-right example illustrates how complex 3D asset structures can cause texture blending issues, particularly where intricate geometry is present.
There are several ways to reduce the influences from those factors. First, choosing 3D assets with reasonable number of meshes, and with plausible textures may filter the low-quality and texture blending failures. Second, using simple and compact image description prompts to generate long-shot camera-shot images will significantly reduce the duplicate subject failures. Reduce the image generation with high elevation angle from bottom view will significantly reduce physical anomaly.
User Study Details
The Figure G4 shows the UI page of our user study. We randomly selected 200 images from the Ultimate3D dataset. Each image corresponds to an RGB prior generated by Blender. During the user study, users will see a pair of side-by-side images. One is an RGB image generated by Blender, and the other is the synthetic image generated by DMs using that RGB image as guidance prior. A vivid preserving of 3D geometry and structure in the synthetic image is a success. Users are required to provide a binary answer regarding the success of the image generation. Each image pair will be manually reviewed by at least 3 users.
Additional Qualitative Results
Based on Ultimate3D benchmark, we provide additional Figure H6 to show more qualitative comparisons between finetuned LLaVA model to commercial SOTAs. The finetuned LLaVA model outperforms other models.

Figure H6. Additional Qualitative Results with Gemma. We evaluate camera-object relation recognition for GPT-5.5, Gemini-3.5-Flash under the high thinking setting, Gemma-4-31B-it, and fine-tuned Gemma-4-31B-it on the Ultimate3D benchmark. Each model is asked questions, shown in gray boxes, about object orientation, camera viewpoint, and camera-shot type, together with the input images on the left. The responses show that fine-tuning Gemma-4-31B-it on Ultimate3D substantially improves camera-object relation recognition compared with the base Gemma-4-31B-it and closed-source models.
Additional Visuals of Ultimate3D Dataset
In Figure L9, we shows more examples of diversity on object categories, camera-object relation, and background contexts.

Figure L9. Diversity of Ultimate3D. Our Ultimate3D dataset and benchmark cover 100 categories of objects, range diverse camera-object relation settings, and provide plausible image quality. (Each row shows images with the same orientation but in diverse subject and context.)
Generalization to Real Images: ImageNet3D
To test whether the camera–object relation knowledge learned from our synthetic data transfers to real photographs, we evaluate on ImageNet3D [90], a benchmark of K ImageNet images with human-annotated 3D object poses across 189 categories, collected independently of our pipeline. We convert its validation split into our VQA format, keeping images that contain a single quality-checked annotated object (23,224 images, 186 categories). Annotated azimuth and elevation are discretized into our eight orientation and three viewpoint classes; we validated this mapping against PASCAL3D+ labels on the 1,109 images shared by both datasets (97.5% and 99.0% agreement, respectively). As ImageNet3D does not annotate camera shot, we derive shot labels from object framing (the maximum bounding-box extent relative to the image), calibrated to our three shot classes, with ambiguous boundary ranges excluded. Since real photographs are strongly biased toward frontal, eye-level views, we sample an answer-balanced evaluation set of 1,200 questions per task. ImageNet3D is used for evaluation only.
Table J2 compares each baseline with its Ultimate3D-fine-tuned counterpart. Fine-tuning improves every model on every task (+15.2, +7.8, and +7.8 points on average for orientation, viewpoint, and camera shot). The baselines largely collapse onto frequent answers (e.g., “Front” for most orientation questions), whereas the fine-tuned models spread predictions across all classes and gain most on rear and oblique views, indicating improved camera–object understanding rather than better answer priors. The one residual gap is the Long-shot class, whose accuracy decreases after fine-tuning; we attribute this to our synthetic long shots being clean, object-centered renders, unlike cluttered real-world wide shots, and leave closing this gap to future work.
| Model | Orientation | Viewpoint | Camera shot |
| Qwen3.5-2B | |||
| Qwen3.5-9B | |||
| Qwen3-VL-8B | |||
| Gemma-4-E2B-it | |||
| Gemma-4-E4B-it | |||
| Gemma-4-26B-A4B-it | |||
| Gemma-4-31B-it | |||
| Avg. gain | +15.2 | +7.8 | +7.8 |
Table J2. Accuracy (%) on the answer-balanced ImageNet3D evaluation set, reported as base Ultimate3D-fine-tuned. Chance is 12.5% for orientation and 33.3% for viewpoint and camera shot.
Failure Analysis
We analyse the errors that remain after Ultimate3D fine-tuning on the synthetic Uti3D benchmark, the real-image PASCAL3D+ and MEBOW splits, and the ImageNet3D set of Appendix J. All ranges below are over the three fine-tuned models.
Which relations remain challenging. Table K3 gives per-class accuracy. On synthetic images accuracy is nearly uniform across classes, so the answer classes themselves are no longer a source of systematic error. On real photographs three relations stay hard: rear-oblique orientations (Back-Left/Back-Right, 42–61% on PASCAL3D+ against 74–85% for the pure side views; Back-Left→Left and Back-Right→Right alone account for 117 of Qwen3.5-9B’s 449 PASCAL3D+ orientation errors), the Bottom viewpoint (41–59% on PASCAL3D+, 25–47% on ImageNet3D, mostly answered Horizontal), and Long-shot framing on real photographs (14–22%), where our clean object-centred renders transfer poorly to cluttered wide shots. People (MEBOW) are the weakest object class overall, and only 2.3% of our synthetic training questions depict humans, none of them ordinary pedestrians.
Table K3: Per-class accuracy (%) after Ultimate3D fine-tuning; each cell is the range over Qwen3.5-2B, Qwen3.5-9B and Qwen3-VL-8B. PASCAL3D+ carries no camera-shot annotation and MEBOW only orientation. Rear-oblique orientations, Bottom and Long-shot are the residual weak classes.
| Relation | Class | Synthetic | PASCAL3D+ | MEBOW | ImageNet3D |
| Orientation | Back | 79–91 | 48–62 | 48–68 | 28–43 |
| Orientation | Back Left | 76–89 | 43–61 | 45–70 | 26–43 |
| Orientation | Left | 87–89 | 74–85 | 33–47 | 35–53 |
| Orientation | Front Left | 77–90 | 54–73 | 29–39 | 24–35 |
| Orientation | Front | 83–89 | 66–71 | 28–34 | 29–39 |
| Orientation | Front Right | 72–85 | 62–69 | 30–39 | 20–48 |
| Orientation | Right | 87–88 | 75–85 | 46–58 | 40–41 |
| Orientation | Back Right | 71–93 | 42–57 | 28–47 | 13–37 |
| Orientation | Overall | 82–86 | 59–64 | 37–38 | 32–34 |
| Viewpoint | Top | 70–85 | 52–64 | – | 42–61 |
| Viewpoint | Horizontal | 63–91 | 66–76 | – | 47–59 |
| Viewpoint | Bottom | 57–72 | 41–59 | – | 25–47 |
| Viewpoint | Overall | 67–83 | 58–62 | – | 45–50 |
| Camera shot | Close-up | 89–96 | – | – | 82–90 |
| Camera shot | Medium-shot | 83–88 | – | – | 40–52 |
| Camera shot | Long-shot | 88–91 | – | – | 14–22 |
| Camera shot | Overall | 87–92 | – | – | 49–52 |
Table K3.
Residual errors are one bin off. Fine-tuning changes the shape of the errors, not just their number: on PASCAL3D+ the share of orientation errors landing in a bin adjacent to the truth rises from 38–55% (base) to 63–64%, while gross () errors fall from 15–30% to 8–15%. Tolerating a one-bin error raises PASCAL3D+ accuracy from 59–64% to 85–87% and synthetic accuracy from 82–86% to 91–93% (base models reach only 55–67% and 69–75% under the same tolerance). A second mode sits at (20–22% of synthetic and 8–16% of PASCAL3D+ orientation errors) and concentrates on objects whose silhouette hides front from back — fans, monitors, sunglasses, and boats, buses or trains seen end-on (Figure K7, row 2).

Figure K7. Representative failures, part 1 of 2: geometric cues in the image plane. Every image was answered incorrectly by all three fine-tuned models; panels give the ground-truth label (green) and the answer of Qwen3.5-9B + Ultimate3D (red). Boundary: the true azimuth is within a few degrees of a bin edge. Symmetry: front and back are barely distinguishable and the prediction is inverted. Rear-oblique: a shallow rear three-quarter view is answered with the neighbouring side bin. Elevation, framing and people failures are shown in Figure K8.

Figure K8. Fig. K8: Representative failures, part 2 of 2: viewpoint, framing and people; continued from Figure K7. Every image was answered incorrectly by all three fine-tuned models; panels give the ground-truth label (green) and the answer of Qwen3.5-9B + Ultimate3D (red). Elevation: the camera is above or below but the cue is faint. Framing: the object fills the frame uncropped, at the Close-up/Medium-shot threshold. People: clutter, small scale, or a head facing away from the torso.
Errors concentrate at the label boundaries. Our benchmark discretises continuous camera parameters, and the residual errors are largely poses near a class boundary: using the exact render parameters, synthetic accuracy falls to 69–71% for orientations within of a bin edge, to 54–59% for elevations within of the band edge and to 36–44% for Medium-shots within 0.15 of the Close-up threshold, against 70–93% away from those boundaries; on ImageNet3D, whose azimuths are human-annotated, accuracy falls from 39–41% to 25–28% within of a boundary, and from 35–37% to 15–20% for objects covering less than 30% of the frame.
Image conditions. We annotated 456 images: per (source, relation) group, 30 that all three finetuned models answered incorrectly, 20 answered incorrectly by one or two of them, and 20 answered correctly by all three as a matched control. Two annotators labelled every image independently and blind to all model predictions, with 80–99% agreement per field. Table K4 compares the shared failures with the matched successes. Failures are enriched for subtle geometric cues, near-boundary poses, clutter, small objects and shape-symmetric objects, but not for occlusion or truncation (33% vs. 32% on real photographs): partial visibility is not the limiting factor, cue strength is. Person images behave differently, with clutter the
| Real photographs | Synthetic | |||
| Condition | F | S | F | S |
| Subtle directional / elevation cue | 52 | 27 | 41 | 28 |
| Pose near a class boundary | 47 | 28 | 41 | 27 |
| Cluttered background | 37 | 18 | 18 | 13 |
| Small object (< 15% of frame) | 24 | 15 | 36 | 20 |
| Shape-symmetric object | 9 | 2 | 6 | 3 |
| Unusual or articulated pose | 13 | 12 | 17 | 13 |
| Multiple instances of the category | 12 | 10 | 4 | 7 |
| Occluded or truncated by the frame | 33 | 32 | 23 | 43 |
| Ground-truth label judged questionable | 38 | 0 | 34 | 15 |
Table K4. Prevalence (%) of image conditions among images all three fine-tuned models answered incorrectly (F) and matched images all three answered correctly (S); and per column pair. Real = PASCAL3D+ and MEBOW.

Figure H5. Additional Qualitative Results with LLaVA. We evaluate camera-object relation recognition for GPT-4o, Claude-3-Sonnet, LLaVA-1.6-13B, and finetuned LLaVA-1.6-13B on the Ultimate3D benchmark. Each model is asked questions, shown in gray boxes, about object orientation, camera viewpoint, and camera-shot type, together with the input images on the left. The responses show that finetuning LLaVA-1.6-13B on Ultimate3D substantially improves camera-object relation recognition compared with the base LLaVA model and closed-source models.
strongest signal (43% vs. 0%) alongside head–torso direction conflicts — conditions absent from our single-object renders.
Label ambiguity. The last row of Table K4 is a limitation of the benchmarks. On orientation the annotators reproduced the ground-truth label for 83–93% of the shared failures, so those are genuine model errors; on viewpoint and camera shot only 43–70%, because the discretisation admits no visual resolution ( of elevation is Top, is Horizontal). We therefore also report adjacent-bin accuracy for orientation and note the elevation band and the fixed distance thresholds as design choices whose boundaries are inherently ambiguous.
Additional Discussions
We provide some insights of using synthetic generated visual instruction dataset for MLLM fine-tuning. For the task of camera-object relation recognition for 100 categories (as we collected in Ultimate3D dataset), a dataset comprising 100K to 1M VQAs will significantly improve baseline MLLMs by one-epoch finetuning. Further scaling up the dataset volume, finetuneing epochs, or model parameters may yield saturation.
Moreover, the quality of synthetic dataset is crucial to the improvement margin. Using dataset with higher success rate (ours is 93.07% as main paper § 4.7) provides better performances than using dataset generated other alternative backbones (i.e. SD-V1.5).
The coverage of arbitrary camera-object relation is also important for generalization. Our preliminary test using real images which are highly biased to front-facing direction from Pascal3D+ [86], results in poor generalization on uncovered orientations and viewpoints. This illustrates the importance of using synthetic dataset to eliminate the dataset biases.
For numerical prediction of camera-object relation, our preliminary test shows plausible success using MEBOW [85] dataset to improve MLLMs on predicting human orientation angles. The average degree prediction accuracy is competitive to the original paper [85]. However, the extended numerical prediction of 100 categories and 3 types of camera-object relations may require even larger dataset volume than Ultimate3D. Our framework is able to scale up the dataset volume. But due to the limitation of computing power, we leave it as interesting future work.
Algorithm of 3D visual instruction dataset generation pipeline
Algorithm 1 Synthetic VQA generation
Input: 3D asset , asset category , camera-object relation .
Parameter: Renderer , DM-based image generator , image decoder , LLM text generator , system prompt given to for generating image context description , and for generating QA pairs , DM sampling steps .
Output: Generated synthetic image , corresponding text QA pairs .
\hspace{1em} # Render 3D priors
\hspace{1em} # Generate image description
for do
\hspace{1em} # DM denoising
end for
\hspace{1em} # Generate QA pairs
return
References
- [1]Dai, W., Li, J., Li, D., Tiong, A., Zhao, J., Wang, W., Li, B., Fung, P.N., Hoi, S.: Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in neural information processing systems 36, 49250–49267 (2023)arxiv.org/abs/2305.06500
- [2]Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In: International Conference on Machine Learning, pp. 19730–19742 (2023). PMLRarxiv.org/abs/2301.12597
- [3]Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. Advances in neural information processing systems 36 (2024)arxiv.org/abs/2304.08485
- [4]Liu, H., Li, C., Li, Y., Lee, Y.J.: Improved baselines with visual instruction tuning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26296–26306 (2024)arxiv.org/abs/2310.03744
- [5]Li, F., Zhang, R., Zhang, H., Zhang, Y., Li, B., Li, W., Ma, Z., Li, C.: Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895 (2024)arxiv.org/abs/2407.07895
- [6]Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al.: Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185–24198 (2024)
- [7]Lin, J., Yin, H., Ping, W., Molchanov, P., Shoeybi, M., Han, S.: Vila: On pre-training for visual language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26689–26699 (2024)arxiv.org/abs/2312.07533
- [8]Tong, S., Brown, E., Wu, P., Woo, S., Middepogu, M., Akula, S.C., Yang, J., Yang, S., Iyer, A., Pan, X., et al.: Cambrian-1: A fully open, vision-centric exploration of multi-modal llms. arXiv preprint arXiv:2406.16860 (2024)arxiv.org/abs/2406.16860
- [9]Meta: Llama 3.2 Vision. https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/ (2024)
- [10]Google DeepMind: Gemma 4 Model Card. https://ai.google.dev/gemma/docs/core/model_card_4. Accessed: 2026-05-17 (2026)
- [11]Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., et al.: Qwen3-vl technical report. arXiv preprint arXiv:2511.21631 (2025)arxiv.org/abs/2511.21631
- [12]Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al.: Qwen3 technical report. arXiv preprint arXiv:2505.09388 (2025)arxiv.org/abs/2505.09388
- [13]Dosovitskiy, A.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)arxiv.org/abs/2010.11929
- [14]Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al.: Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023)arxiv.org/abs/2304.07193
- [15]Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J.E., Stoica, I., Xing, E.P.: Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90% ChatGPT Quality (2023). https://lmsys.org/blog/2023-03-30-vicuna/
- [16]Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)arxiv.org/abs/2302.13971
- [17]Google DeepMind: Gemini. https://deepmind.google/models/gemini/. Accessed: 2026-05-22 (2026) Singh, A., Fry, A., Perelman, A., Tart, J., Ganesh, A., El-Kishky, A., McLaughlin, A., Low, A., Ostrow, A., Anantharam, A., Nathan, A., Luo, A., Helyar, A., Madry, A., Efremov, A., Spyra, A., Baker-Whitcomb, A., Beutel, A., Karpenko, A., Makelov, A., Neitz, A., Wei, A., Barr, A., Kirchmeyer, A., Ivanov, A., Christakis, A., Gillespie, A., Tam, A., Bennett, A., Wan, A., Huang, A., Sandjideh, A.M., Yang, A., Kumar, A., Saraiva, A., Vallone, A., Gheorghe, A., Garcia, A.G., Braunstein, A., Liu, A., Schmidt, A., Mereskin, A., Mishchenko, A., Applebaum, A., Rogerson, A., Rajan, A., Wei, A., Kotha, A., Srivastava, A., Agrawal, A., Vijavvergiya, A., Tyra, A., Nair, A., Nayak, A., Eggers, B., Ji, B., Hoover, B., Chen, B., Chen, B., Barak, B., Minaiev, B., Hao, B., Baker, B., Lightcap, B., McKinzie, B., Wang, B., Quinn, B., Fioca, B., Hsu, B., Yang, B., Yu, B., Zhang, B., Brenner, B., Zetino, C.R., Raymond, C., Lugaresi, C., Paz, C., Hudson, C., Whitney, C., Li, C., Chen, C., Cole, C., Voss, C., Ding, C., Shen, C., Huang, C., Colby, C., Hallacy, C., Koch, C., Lu, C., Kaplan, C., Kim, C., Minott-Henriques, C., Frey, C., Yu, C., Czarnicki, C., Reid, C., Wei, C., Decareaux, C., Scheau, C., Zhang, C., Forbes, C., Tang, D., Goldberg, D., Roberts, D., Palmie, D., Kappler, D., Levine, D., Wright, D., Leo, D., Lin, D., Robinson, D., Grabb, D., Chen, D., Lim, D., Salama, D., Bhattacharjee, D., Tsipras, D., Li, D., Yu, D., Strouse, D., Williams, D., Hunn, D., Bayes, E., Arbus, E., Akyurek, E., Le, E.Y., Widmann, E., Yani, E., Proehl, E., Sert, E., Cheung, E., Schwartz, E., Han, E., Jiang, E., Mitchell, E., Sigler, E., Wallace, E., Ritter, E., Kavanaugh, E., Mays, E., Nikishin, E., Li, F., Such, F.P., Avila Belbute Peres, F., Raso, F., Bekerman, F., Tsimpourlas, F., Chatzis, F., Song, F., Zhang, F., Raila, G., McGrath, G., Briggs, G., Yang, G., Parascandolo, G., Chabot, G., Kim, G., Zhao, G., Valiant, G., Leclerc, G., Salman, H., Wang, H., Sheng, H., Jiang, H., Wang, H., Jin, H., Sikchi, H., Schmidt, H., Aspegren, H., Chen, H., Qiu, H., Lightman, H., Covert, I., Kivlichan, I., Silber, I., Sohl, I., Hammoud, I., Clavera, I., Lan, I., Akkaya, I., Kostrikov, I., Kofman, I., Ettinger, I., Singal, I., Hehir, J., Huh, J., Pan, J., Wilczynski, J., Pachocki, J., Lee, J., Quinn, J., Kiros, J., Kalra, J., Samaroo, J., Wang, J., Wolfe, J., Chen, J., Wang, J., Harb, J., Han, J., Wang, J., Zhao, J., Chen, J., Yang, J., Tworek, J., Chand, J., Landon, J., Liang, J., Lin, J., Liu, J., Wang, J., Tang, J., Yin, J., Jang, J., Morris, J., Flynn, J., Ferstad, J., Heidecke, J., Fishbein, J., Hallman, J., Grant, J., Chien, J., Gordon, J., Park, J., Liss, J., Kraaijeveld, J., Guay, J., Mo, J., Lawson, J., McGrath, J., Vendrow, J., Jiao, J., Lee, J., Steele, J., Wang, J., Mao, J., Chen, K., Hayashi, K., Xiao, K., Salahi, K., Wu, K., Sekhri, K., Sharma, K., Singhal, K., Li, K., Nguyen, K., Gu-Lemberg, K., King, K., Liu, K., Stone, K., Yu, K., Ying, K., Georgiev, K., Lim, K., Tirumala, K., Miller, K., Ahmad, L., Lv, L., Clare, L., Fauconnier, L., Itow, L., Yang, L., Romaniuk, L., Anise, L., Byron, L., Pathak, L., Maksin, L., Lo, L., Ho, L., Jing, L., Wu, L., Xiong, L., Mamit-suka, L., Yang, L., McCallum, L., Held, L., Bourgeois, L., Engstrom, L., Kuhn, L., Feurvrier, L., Zhang, L., Switzer, L., Kondraciuk, L., Kaiser, L., Joglekar, M., Singh, M., Shah, M., Stratta, M., Williams, M., Chen, M., Sun, M., Cayton, M., Li, M., Zhang, M., Aljubeh, M., Nichols, M., Haynes, M., Schwarzer, M., Gupta, M., Shah, M., Guan, M.Y., Huang, M., Dong, M., Wang, M., Glaese, M., Carroll, M., Lampe, M., Malek, M., Sharman, M., Zhang, M., Wang, M., Pokrass, M., Florian, M., Pavlov, M., Wang, M., Chen, M., Wang, M., Feng, M., Bavarian, M., Lin, M., Abdool, M., Rohaninejad, M., Soto, N., Staudacher, N., LaFontaine, N., Marwell, N., Liu, N., Preston, N., Turney, N., Ansman, N., Blades, N., Pancha, N., Mikhaylin, N., Felix, N., Handa, N., Rai, N., Keskar, N., Brown, N., Nachum, O., Boiko, O., Murk, O., Watkins, O., Gleeson, O., Mishkin, P., Lesiewicz, P., Baltescu, P., Belov, P., Zhokhov, P., Pronin, P., Guo, P., Thacker, P., Liu, Q., Yuan, Q., Liu, Q., Dias, R., Puckett, R., Arora, R., Mullapudi, R.T., Gaon, R., Miyara, R., Song, R., Agarwal, R., Marsan, R., Yemiuri, R., Xiong, R., Kshirsagar, R., Nuttall, R., Tsiupa, R., Eldan, R., Wang, R., James, R., Ziv, R., Shu, R., Nigmatullin, R., Jain, S., Talaie, S., Altman, S., Arnesen, S., Toizer, S., Toyer, S., Misenendino, S., Agarwal, S., Yoo, S., Heon, S., Ethersmith, S., Grove, S., Taylor, S., Bubeck, S., Banesiu, S., Amdo, S., Zhao, S., Wu, S., Santurkar, S., Zhao, S., Chaudhuri, S.R., Krishnaswamy, S., Shuaigi, Xia, Chen, S., Andkat, S., Fishman, S.P., Tobin, S., Fu, S., Jain, S., Mei, S., Egoian, S., Kim, S., Golden, S., Mah, S., Lin, S., Imm, S., Sharpe, S., Yadlowsky, S., Choudhry, S., Eum, S., Sanjeev, S., Khan, T., Stramer, T., Wang, T., Xin, T., Gogineni, T., Christianson, T., Sanders, T., Patwardhan, T., Degry, T., Shadwell, T., Fu, T., Gao, T., Garipov, T., Sriskandarajah, T., Sherbakov, T., Korbak, T., Kaftan, T., Hiratsuka, T., Wang, T., Song, T., Zhao, T., Peterson, T., Kharitonov, V., Chernova, V., Kosaraju, V., Kuo, V., Pong, V., Verma, V., Petrov, V., Jiang, W., Zhang, W., Zhou, W., Xie, W., Zhang, W., McCabe, W., DePue, W., Ellsworth, W., Bain, W., Thompson, W., Chen, X., Qi, X., Xiang, X., Shi, X., Dubois, Y., Yu, Y., Khakbaz, Y., Wu, Y., Qian, Y., Lee, Y.T., Chen, Y., Zhang, Y., Xiong, Y., Tian, Y., Chen, Y., Cha, Y., Bai, Y., Yang, Y., Yuan, Y., Li, Y., Zhang, Y., Yang, Y., Jin, Y., Jiang, Y., Wang, Y., Wang, Y., Liu, Y., Stubenvoll, Z., Dou, Z., Wu, Z., Wang, Z.: OpenAI GPT-5 System Card (2026). https://arxiv.org/abs/2601.03267
- [19]Anthropic: Introducing Claude Opus 4.7. https://www.anthropic.com/news/claude-opus-4-7. Accessed: 2026-05-22 (2026)
- [20]Yang, Z., Li, L., Lin, K., Wang, J., Lin, C.-C., Liu, Z., Wang, L.: The dawn of lmms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309.17421 9(1), 1 (2023)arxiv.org/abs/2309.17421
- [21]Tong, S., Liu, Z., Zhai, Y., Ma, Y., LeCun, Y., Xie, S.: Eyes wide shut? exploring the visual shortcomings of multimodal llms. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9568–9578 (2024)arxiv.org/abs/2401.06209
- [22]Fu, X., Hu, Y., Li, B., Feng, Y., Wang, H., Lin, X., Roth, D., Smith, N.A., Ma, W.-C., Krishna, R.: Blink: Multimodal large language models can see but not perceive. arXiv preprint arXiv:2404.12390 (2024)arxiv.org/abs/2404.12390
- [23]Yuksekgonul, M., Bianchi, F., Kalluri, P., Jurafsky, D., Zou, J.: When and why vision-language models behave like bags-of-words, and what to do about it? arXiv preprint arXiv:2210.01936 (2022)
- [24]Cheng, A.-C., Yin, H., Fu, Y., Guo, Q., Yang, R., Kautz, J., Wang, X., Liu, S.: Spatialrgpt: Grounded spatial reasoning in vision language model. arXiv preprint arXiv:2406.01584 (2024)arxiv.org/abs/2406.01584
- [25]Chen, B., Xu, Z., Kirmani, S., Ichter, B., Sadigh, D., Guibas, L., Xia, F.: Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14455–14465 (2024)arxiv.org/abs/2401.12168
- [26]Guo, Q., De Mello, S., Yin, H., Byeon, W., Cheung, K.C., Yu, Y., Luo, P., Liu, S.: Regiongpt: Towards region understanding vision language model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13796–13806 (2024)arxiv.org/abs/2403.02330
- [27]Chen, G.H., Chen, S., Zhang, R., Chen, J., Wu, X., Zhang, Z., Chen, Z., Li, J., Wan, X., Wang, B.: Allava: Harnessing gpt4v-synthesized data for a lite vision-language model. arXiv preprint arXiv:2402.11684 (2024)arxiv.org/abs/2402.11684
- [28]Li, X., Tu, H., Hui, M., Wang, Z., Zhao, B., Xiao, J., Ren, S., Mei, J., Liu, Q., Zheng, H., et al.: What if we recaption billions of web images with llama-3? arXiv preprint arXiv:2406.08478 (2024)arxiv.org/abs/2406.08478
- [29]Zhang, Y., Wu, J., Li, W., Li, B., Ma, Z., Liu, Z., Li, C.: Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713 (2024)arxiv.org/abs/2410.02713
- [30]Sharifzadeh, S., Kaplanis, C., Pathak, S., Kumaran, D., Ilic, A., Mitrovic, J., Blundell, C., Banino, A.: Synth2: Boosting visual-language models with synthetic captions and image embeddings. arXiv preprint arXiv:2403.07750 (2024)arxiv.org/abs/2403.07750
- [31]Taylor, S., Bubeck, S., Banesiu, S., Amdo, S., Zhao, S., Wu, S., Santurkar, S., Zhao, S., Chaudhuri, S.R., Krishnaswamy, S., Shuaigui, Xia, Chen, S., Andkdat, S., Fishman, S.P., Tobin, S., Fu, S., Jain, S., Mei, S., Egoian, S., Kim, S., Golden, S., Mah, S., Lin, S., Imm, S., Sharpe, S., Yadlowsky, S., Choudhry, S., Eum, S., Sanjeev, S., Khan, T., Stramer, T., Wang, T., Xin, T., Gogineni, T., Christianson, T., Sanders, T., Patwardhan, T., Degry, T., Shadwell, T., Fu, T., Gao, T., Garipov, T., Sriskandarajah, T., Sherbakov, T., Korbak, T., Kaftan, T., Hiratsuka, T., Wang, T., Song, T., Zhao, T., Peterson, T., Kharitonov, V., Chernova, V., Kosaraju, V., Kuo, V., Pong, V., Verma, V., Petrov, V., Jiang, W., Zhang, W., Zhou, W., Xie, W., Zhan, W., McCabe, W., DePue, W., Ellsworth, W., Bain, W., Thompson, W., Chen, X., Qi, X., Xiang, X., Shi, X., Dubois, Y., Yu, Y., Khakbaz, Y., Wu, Y., Qian, Y., Lee, Y.T., Chen, Y., Zhang, Y., Xiong, Y., Tian, Y., Cha, Y., Bai, Y., Yang, Y., Yuan, Y., Li, Y., Zhang, Y., Yang, Y., Jin, Y., Jiang, Y., Wang, Y., Wang, Y., Liu, Y., Stubenvoll, Z., Dou, Z., Wu, Z., Wang, Z.: OpenAI GPT-5 System Card (2026). https://arxiv.org/abs/2601.03267 Zhang, Y., Zhang, R., Gu, J., Zhou, Y., Lipka, N., Yang, D., Sun, T.: Llava\ r: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint arXiv:2306.17107 (2023)
- [32]Du, Y., Guo, H., Zhou, K., Zhao, W.X., Wang, J., Wang, C., Cai, M., Song, R., Wen, J.-R.: What makes for good visual instructions? synthesizing complex visual reasoning instructions for visual instruction tuning. arXiv preprint arXiv:2311.01487 (2023)arxiv.org/abs/2311.01487
- [33]Liu, Z., Liang, H., Xiong, W., Yu, Q., He, C., Cui, B., Zhang, W.: Synthvlm: High-efficiency and high-quality synthetic data for vision language models. arXiv preprint arXiv:2407.20756 (2024)
- [34]Du, Z., Li, H., Guo, X., Li, B.: Training on synthetic data beats real data in multimodal relation extraction. arXiv preprint arXiv:2312.03025 (2023)arxiv.org/abs/2312.03025
- [35]Li, Y., Zhang, C., Yu, G., Wang, Z., Fu, B., Lin, G., Shen, C., Chen, L., Wei, Y.: Stablellava: Enhanced visual instruction tuning with synthesized image-dialogue data. arXiv preprint arXiv:2308.10253 (2023)arxiv.org/abs/2308.10253
- [36]Johnson, J., Hariharan, B., Van Der Maaten, L., Fei-Fei, L., Lawrence Zitnick, C., Girshick, R.: Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2901–2910 (2017)arxiv.org/abs/1612.06890
- [37]Liu, R., Liu, C., Bai, Y., Yuille, A.L.: Clevr-ref+: Diagnosing visual reasoning with referring expressions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4185–4194 (2019)arxiv.org/abs/1901.00850
- [38]Greff, K., Belletti, F., Beyer, L., Doersch, C., Du, Y., Duckworth, D., Fleet, D.J., Gnanapragasam, D., Golemo, F., Herrmann, C., et al.: Kubric: A scalable dataset generator. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3749–3761 (2022)arxiv.org/abs/2203.03570
- [39]Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740–755 (2014). Springerarxiv.org/abs/1405.0312
- [40]Young, P., Lai, A., Hodosh, M., Hockenmaier, J.: From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics 2, 67–78 (2014)DOI
- [41]Yu, L., Poirson, P., Yang, S., Berg, A.C., Berg, T.L.: Modeling context in referring expressions. In: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pp. 69–85 (2016). Springerarxiv.org/abs/1608.00272
- [42]Hudson, D.A., Manning, C.D.: Gqa: A new dataset for real-world visual reasoning and compositional question answering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6700–6709 (2019)arxiv.org/abs/1902.09506
- [43]Liu, F., Emerson, G., Collier, N.: Visual spatial reasoning. Transactions of the Association for Computational Linguistics 11, 635–651 (2023)arxiv.org/abs/2205.00363
- [44]Gurari, D., Li, Q., Stangl, A.J., Guo, A., Lin, C., Grauman, K., Luo, J., Bigham, J.P.: Vizwiz grand challenge: Answering visual questions from blind people. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3608–3617 (2018)arxiv.org/abs/1802.08218
- [45]Wang, J., Meng, L., Weng, Z., He, B., Wu, Z., Jiang, Y.-G.: To see is to believe: Prompting gpt-4v for better visual instruction tuning. arXiv preprint arXiv:2311.07574 (2023)arxiv.org/abs/2311.07574
- [46]Marino, K., Rastegari, M., Farhadi, A., Mottaghi, R.: Ok-vqa: A visual question answering benchmark requiring external knowledge. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3195–3204 (2019)arxiv.org/abs/1906.00067
- [47]Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., Kalyan, A.: Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems 35, 2507–2521 (2022)arxiv.org/abs/2209.09513
- [48]Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., Parikh, D.: Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6904–6913 (2017)arxiv.org/abs/1612.00837
- [49]Mishra, A., Shekhar, S., Singh, A.K., Chakraborty, A.: Ocr-vqa: Visual question answering by reading text in images. In: 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 947–952 (2019). IEEEDOI
- [50]Singh, A., Natarjan, V., Shah, M., Jiang, Y., Chen, X., Parikh, D., Rohrbach, M.: Towards vqa models that can read. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8317–8326 (2019)arxiv.org/abs/1904.08920
- [51]Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al.: Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems 35, 25278–25294 (2022)
- [52]Gupta, A., Dollar, P., Girshick, R.: Lvis: A dataset for large vocabulary instance segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5356–5364 (2019)arxiv.org/abs/1908.03195
- [53]Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Krawitz, J., Chen, S., Kalanditis, Y., Li, L.-J., Shamma, D.A., et al.: Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision 123, 32–73 (2017)arxiv.org/abs/1602.07332
- [54]Zhou, B., Zhao, H., Puig, X., Xiao, T., Fidler, S., Barriuso, A., Torralba, A.: Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision 127(3), 302–321 (2019)arxiv.org/abs/1608.05442
- [55]OpenAI: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)arxiv.org/abs/2303.08774
- [56]Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in neural information processing systems 33, 6840–6851 (2020)arxiv.org/abs/2006.11239
- [57]Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 (2020)arxiv.org/abs/2010.02502
- [58]Song, Y., He, L., Zhang, Z., Kim, S.Y., Zhang, H., Xiong, W., Lin, Z., Price, B.L., Cohen, S., Zhang, J., et al.: Refine-by-align: Reference-guided artifacts refinement through semantic alignment. In: The Thirteenth International Conference on Learning Representations (2025)arxiv.org/abs/2412.00306
- [59]Song, Y., Zhang, Z., Lin, Z., Cohen, S., Price, B., Zhang, J., Kim, S.Y., Zhang, H., Xiong, W., Aliaga, D.: Imprint: Generative object compositing by learning identity-preserving representation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8048–8058 (2024)arxiv.org/abs/2403.10701
- [60]He, L., Lu, Y., Corring, J., Florencio, D., Zhang, C.: Diffusion-based document layout generation. In: International Conference on Document Analysis and Recognition, pp. 361–378 (2023). SpringerDOI
- [61]He, L., Song, Y., Huang, H., Liu, P., Tang, Y., Aliaga, D., Zhou, X.: Kubrick: Multimodal agent collaborations for synthetic video generation. arXiv preprint arXiv:2408.10453 (2024)arxiv.org/abs/2408.10453
- [62]Song, Y., Zhang, Z., Lin, Z., Cohen, S., Price, B., Zhang, J., Kim, S.Y., Aliaga, D.: Object-stitch: Object compositing with diffusion model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18310–18319 (2023)
- [63]Liao, Y.-H., Elflein, S., He, L., Leal-Taixé, L., Choi, Y., Fidler, S., Acuna, D.: Longperceptualthoughts: Distilling system-2 reasoning for system-1 perception. arXiv preprint arXiv:2504.15362 (2025)arxiv.org/abs/2504.15362
- [64]Hua, H., Zeng, Z., Song, Y., Tang, Y., He, L., Aliaga, D., Xiong, W., Luo, J.: Mmig-bench: Towards comprehensive and explainable evaluation of multi-modal image generation models. arXiv preprint arXiv:2505.19415 (2025)arxiv.org/abs/2505.19415
- [65]Sun, L., He, L., Jia, S., He, Y., You, C.: Docagent: An agentic framework for multi-modal long-context document understanding. In: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 17712–17727 (2025)
- [66]Tang, Y., Bi, J., Huang, C., Liang, S., Shimada, D., Hua, H., Xiao, Y., Song, Y., Liu, P., Feng, M., et al.: Caption anything in video: Fine-grained object-centric captioning via spatiotemporal multimodal prompting. arXiv preprint arXiv:2504.05541 (2025)arxiv.org/abs/2504.05541
- [67]He, L., Aliaga, D.: Coho: Context-sensitive city-scale hierarchical urban layout generation. In: European Conference on Computer Vision, pp. 1–18 (2024). Springer
- [68]Tang, Y., Guo, J., Liu, P., Wang, Z., Hua, H., Zhong, J.-X., Xiao, Y., Huang, C., Song, L., Liang, S., et al.: Generative ai for cel-animation: A survey. arXiv preprint arXiv:2501.06250 (2025)arxiv.org/abs/2501.06250
- [69]He, L., Aliaga, D.: Globalmapper: Arbitrary-shaped urban layout generation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 454–464 (2023)arxiv.org/abs/2307.09693
- [70]He, L., Shan, J., Aliaga, D.: Generative building feature estimation from satellite images. IEEE Transactions on Geoscience and Remote Sensing 61, 1–13 (2023)
- [71]Yuan, Y., Wang, X., Wickremasinghe, T., Nadir, Z., Ma, B., Chan, S.H.: Newton-gen: Physics-consistent and controllable text-to-video generation via neural newtonian dynamics. arXiv preprint arXiv:2509.21309 (2025)
- [72]Yuan, Y., Wang, X., Sheng, Y., Chennuri, P., Zhang, X., Chan, S.: Generative photography: Scene-consistent camera control for realistic text-to-image synthesis. In: Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 7920–7930 (2025)
- [73]Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684–10695 (2022)
- [74]Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., Rombach, R.: Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 (2023)
- [75]Krojer, B., Vattikonda, D., Lara, L., Jampani, V., Portelance, E., Pal, C., Reddy, S.: Learning action and reasoning-centric image editing from videos and simulations. arXiv preprint arXiv:2407.03471 (2024)
- [76]Wang, S., Saharia, C., Montgomery, C., Pont-Tuset, J., Noy, N.S., Pellegrini, S., Onofri, Y., Laszlo, S., Fleet, D.J., Soricut, R., et al.: Imagen editor and editbench: Advancing and evaluating text-guided image inpainting. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18359–18369 (2023)
- [77]Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., Cohen-Or, D.: Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626 (2022)
- [78]Zhang, K., Mo, L., Chen, W., Sun, H., Su, Y.: Magicbrush: A manually annotated dataset for instruction-guided image editing. Advances in Neural Information Processing Systems 36 (2024)
- [79]Zhang, L., Rao, A., Agrawala, M.: Adding conditional control to text-to-image diffusion models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836–3847 (2023)arxiv.org/abs/2302.05543
- [80]Ma, W., Liu, Q., Wang, J., Wang, A., Yuan, X., Zhang, Y., Xiao, Z., Zhang, G., Lu, B., Duan, R., et al.: Generating images with 3d annotations using diffusion models. In: The Twelfth International Conference on Learning Representations (2024)arxiv.org/abs/2306.08103
- [81]Deitke, M., Schwenk, D., Salvador, J., Weihs, L., Michel, O., Vanderbilt, E., Schmidt, L., Ehsani, K., Kembhavi, A., Farhadi, A.: Objaverse: A universe of annotated 3d objects. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13142–13153 (2023)arxiv.org/abs/2212.08051
- [82]Deitke, M., Liu, R., Wallingford, M., Ngo, H., Michel, O., Kusupati, A., Fan, A., Laforte, C., Voleti, V., Gadre, S.Y., et al.: Objaverse-XL: A universe of 10m+ 3d objects. Advances in Neural Information Processing Systems 36 (2024)arxiv.org/abs/2307.05663
- [83]Chang, A.X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., et al.: Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012 (2015)arxiv.org/abs/1512.03012
- [84]Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.: Imagenet large scale visual recognition challenge. International journal of computer vision 115, 211–252 (2015)arxiv.org/abs/1409.0575
- [85]Wu, C., Chen, Y., Luo, J., Su, C.-C., Dawane, A., Hanzra, B., Deng, Z., Liu, B., Wang, J.Z., Kuo, C.-h.: Mebow: Monocular estimation of body orientation in the wild. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3451–3461 (2020)arxiv.org/abs/2011.13688
- [86]Xiang, Y., Mottaghi, R., Savarese, S.: Beyond pascal: A benchmark for 3d object detection in the wild. In: IEEE Winter Conference on Applications of Computer Vision (WACV) (2014)
- [87]Xu, J., Liu, X., Wu, Y., Tong, Y., Li, Q., Ding, M., Tang, J., Dong, Y.: ImageReward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems 36 (2024)arxiv.org/abs/2304.05977
- [88]Hessel, J., Holtzman, A., Forbes, M., Bras, R.L., Choi, Y.: Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718 (2021)arxiv.org/abs/2104.08718
- [89]Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems 30 (2017)arxiv.org/abs/1706.08500
- [90]Ma, W., Zeng, G., Zhang, G., Liu, Q., Zhang, L., Kortylewski, A., Liu, Y., Yuille, A.: Imagenet3d: Towards general-purpose object-level 3d understanding. arXiv preprint arXiv:2406.09613 (2024)arxiv.org/abs/2406.09613
