betaFeedback

WorksComputer Science

Ultimate3D: Teaching Multimodal LLMs 3D Camera-Object Relations by Large-Scale Visual Instruction Data

Abstract

Multimodal Large Language Models (MLLMs) struggle with accurately capturing camera-object relations, especially for object orientation, camera viewpoint, and camera shots. This stems from the fact that existing MLLMs are trained on images with limited diverse camera-object relations and corresponding textual descriptions. To address this, we propose a synthetic generation pipeline to create large-scale 3D visual instruction datasets. Our framework takes 3D assets as input and uses rendering and diffusion-based image generation models to create photorealistic images preserving precise camera-object relations. Additionally, large language models (LLMs) are used to generate text prompts for guiding visual instruction tuning and controlling image generation. We create Ultimate3D, a large-scale 3D-grounded 2D visual instruction dataset. Ultimate3D consists of 240K VQAs with precise camera-object annotations, and corresponding benchmark. MLLMs fine-tuned on our proposed dataset outperform commercial models by a large margin, achieving an average accuracy improvement of 29.7% on camera-object relation recognition tasks. Our code, dataset, and benchmark will contribute to broad MLLM applications.

Introduction

The advent of Multimodal Large Language Models (MLLMs) [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12] have demonstrated remarkable success on tasks including image captioning, embodied robotic manipulation, and visual question answering (VQA). MLLMs bridge visual understanding from vision encoders [13, 14] with powerful verbal capability from large language models (LLMs) [15, 16]. Collectively this provides end-to-end inferencing across image and text modalities. In fact, the recently released Gemini-3.5 [17], GPT-5.5 [18] and Claude-4.7-Opus [19] have extended their performances to be comparable to human intelligence in some of the experimented tasks [20].

However, research works have found needs and weaknesses of MLLMs [21, 22, 23]. Tong et al. [21] point out the deficiency in visual representations learnt by ViT [13]. In particular, two weakness categories (i.e., orientation/direction, viewpoint/perspective) focus on camera-object relations such as distinguishing simple concepts about object orientation (”right”, ”front”, ”back”, etc.), camera viewpoint location (”top”, ”bottom”, etc.), and camera-shot styles (”close-up”, ”long-shot”, etc.). As two examples in Figure 1 show, current SOTA model (i.e., Gemini-3.5) struggle with understanding such camera-object relations. The challenges to improving 3D-aware MLLM abilities on recognizing camera-object relations include: (1) capturing the complexity of the task, and (2) having a dataset containing millions of image-text pairs as is a common requirement for training MLLM on spatial perception tasks [24, 25, 26].

Generating Synthetic Visual Instruction Dataset

Figure 1. Generating Synthetic Visual Instruction Dataset. Our framework uses open-sourced 3D assets to generate photo-realistic images with precisely controlled camera-object relation. Corresponding text instructions are also generated by Large Language Model (LLM). The generated Ultimate3D dataset (240K) and benchmark (8K) advance baseline MLLM models (Gemma-4, Qwen3-VL, etc.) to outperform commercial MLLMs (GPT-5.5, Gemini-3.5, etc.) on camera-object relation recognition tasks. The left panel illustrates representative MLLM failures, while the right panel provides a direct same-image comparison between base and Ultimate3D-fine-tuned models.

Nevertheless, to the best of our knowledge, none of the existing MLLMs or training datasets

focus on the aforementioned camera-object relations. Prior methods for MLLM training data generation can be divided into the following two groups: (1) Relabeling of existing real images. These works focus on generating textual captions by deep learning models given existing images [27, 28, 29, 30, 31, 32]. Specifically, SpatialRGPT [24] and SpatialVLM [25] address spatial perception and reasoning weakness by providing a comprehensive data annotation pipeline leveraging multiple image segmentation, depth estimation, and camera calibration models to produce dense predictions for real images. But, in practice the accuracy of the predicted labels is not high and the dataset size and distribution is constrained by existing image datasets which predominantly only contain ”front” view objects. (2) Generating synthetic images. Typically text-to-image DMs are utilized for synthetic image generation based on provided text prompts [33, 34, 35]. But, their score-based data curation is not robust to camera-object relations, or generation failures (See Appendix § 11). Alternatively, simulation-based generation pipelines for visual reasoning [36, 37, 38] may provide ground truth camera-object relations, but the generated image quality, complexity, and diversity are still far from realistic. In summary, existing synthetic dataset generation pipelines fail to provide image-text pairs with ground truth camera-object relation, realistic image quality, and diverse image subject and context.

In this work, we propose a synthetic generation pipeline to create unlimited image-text pairs focusing on camera-object relations. With 3D assets (e.g., 3D models with texture, materials, and animation) as input, our framework utilizes a renderer (e.g., Blender) with arbitrary camera-object relation parameters to render a series of 3D ground truth priors (i.e., RGB, depth, segmentation). Those 3D priors are used as comprehensive conditions for stacked diffusion-based generation models (e.g., ControlNet) to generate photorealistic images preserving precise camera-object relations. Simultaneously, LLMs are used to generate versatile text prompts for the controlling of image context and object appearance, and to generate diverse VQA prompts regarding camera-object relations. With our framework, we have created the Ultimate3D dataset as a future benchmark for camera-object relations. The proposed framework serves as a data-generation pipeline that produces photorealistic image-text pairs with precisely controlled camera-object relations from 3D assets. We then use standard fine-tuning protocols to evaluate the effectiveness of Ultimate3D across multiple open-source MLLMs, including LLaVA-, Llama-, Gemma-, and Qwen-based models. Models fine-tuned on Ultimate3D outperform commercial MLLMs by a large margin (i.e., an average 29.7% accuracy improvement) on recognizing camera-object relations.

Our contributions include:

  • A synthetic generation pipeline for unlimited image-text pair creation of diverse camera-object relations.

  • A dataset and benchmark for further improving and evaluating camera-object relations in MLLMs.

  • Extensive evaluation across multiple open-source MLLMs, showing that Ultimate3D fine-tuning consistently improves camera-object relation recognition and outperforms commercial SOTA models.

Related Works

Datasets and Benchmarks for MLLMs

MLLMs use various dataset recipes for visual instruction tuning [1, 2, 3, 4, 5, 6, 7, 8]. Typically, dozens of datasets, focusing on different areas, are combined to enable the comprehensive abilities of MLLMs. Datasets include image captioning [39, 40, 41], visual reasoning [42, 43], general VQA data [3, 44, 45] on knowledge grounding [46, 47], image understanding [48], and OCR [49, 50]. Further, most visual instruction datasets are relabeled or created from other existing image-based datasets (e.g., COCO [39], LAION [51], LVIS [52], Visual Genome [53], ADE20K [54]). As far as we know, none of the datasets focus on camera-object relations. Thus, the missing ground truth camera-object relation data exacerbates the task for deep learning models, such as SpatialRGPT [24] and SpatialVLM [25]. In fact, multiple papers have indicated MLLMs’ poor distinguishing ability on camera-object relations [21, 22]. Our paper fills this gap by providing a generation pipeline, dataset, and benchmark.

In particular for the visual instruction data generation, LLMs, like GPT-4o [55], have been successfully utilized in LLaVA [3, 43] for dialogue content instruction generation, and used in MMVP [21] for answer grading. However, generating high-quality images corresponding to proposed text prompts has not been well developed. In this project, we leverage the diverse text generation capability of GPT-4o for the text prompt generation and we provide an image generation pipeline regarding 3D-awareness.

Simulation-based Image Generation

CLEVR [36] and Kubric [38] use 3D engines to generate synthetic images by rendering 3D primitives at multiple arbitrary camera-object relations. Generally, simulation-based image rendering easily preserve ground truth camera-object relations in arbitrary viewpoint settings. However, the generated images are neither photorealistic nor diverse. Realism is limited by the assets’ level of detail and textures, and diversity is restricted by the small number of assets. These limitations constrain the usage of synthetic images in MLLM training to only object counting or simple spatial reasoning tasks [8]. In this project, we leverage the 3D geometry prior rendered by a 3D renderer to precisely guide the image generation pipeline for preserving ground truth camera-object relation.

Generative Images

Generative models provide advanced image generation by iteratively restoring the original data by denoising from Gaussian noise [56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72]. Open-sourced Stable Diffusion [73], and SDXL [74] enable high-resolution and photo-realistic image generation and editing. While generative models provide versatile context and subject, representative works (e.g., AURORA [75], EditBench [76], Prompt-to-Prompt [77], and MagicBrush [78]) fail to control camera-object relations in the generated content, and quality guarantees rely on human checking. ControlNet [79], built on versatile backbone DMs (e.g., SDXL), provides spatial control in the generated images by using various image priors (e.g., Canny edges, semantic segmentation, relative depth).

In particular, Ma et al. [80] explore using synthetic image generation for 3D pose estimation by conditioning diffusion models on pose angles. However, their approach does not address advancing MLLMs 3D spatial understanding, nor does it consider VQA instruction generation. Moreover, their reliance on 2D Canny priors (i.e., silhouette) brings severe artifacts for images with backward viewpoint and complex depth variations. In contrast, our method incorporates depth priors to enhance geometric accuracy and leverages SDXL’s superior fidelity to improve image realism. Further discussions and user studies are in § 4.6 and § 4.7.

Method

Our framework is described in Figure 2 and Appendix § 20. Given input 3D asset AA, its category cc, and arbitrary camera-object relation β\beta, our framework generates photorealistic image IsynI_{syn} preserving β\beta, and corresponding text QA instruction TqaT_{qa}. Our pipeline consists of 3D visual prior rendering (§ 3.1), diverse image description generation (§ 3.2), controlled image generation (§ 3.3), and 3D-aware text instruction generation (§ 3.4).

Our framework

Figure 2. Our Framework. Each shade box corresponds to a section in § 3. Given open-sourced 3D assets, our approach leverages a 3D Renderer to generate 3D visual priors (IβI_{\beta}) preserving ground truth camera-object relation (β\beta). Meanwhile, LLMs take 3D asset category to generate diverse image descriptions (TimgT_{img}) as conditional guidance. Both the 3D visual priors and diverse text prompts are used to generate synthetic images (IsynI_{syn}) by multiple ControlNet-based networks. Corresponding text QA instructions (TqaT_{qa}) are generated by LLMs given the ground truth camera-object relation. Our Ultimate3D dataset and benchmark (i.e. pairs of TqaT_{qa} and IsynI_{syn}) contribute to fine-tuning and evaluation of MLLMs.

3D Visual Prior Rendering

Our system renders several 3D visual priors including 2D depth images, Canny-edge images, and segmentation masks from arbitrary viewpoints. These 3D visual priors contain the scene’s geometry and will be used together with LLM-generated image descriptions to collectively control the DM-based image generation in § 3.3.

Camera-Object Relation Definition. As Figure 3 shows, we define camera-object relations using an explicit 3D asset representation and camera-object relative pose.

Figure 3. 3D Visual Prior Rendering. With a general camera model, our method utilizes a 3D Renderer (e.g., Blender) to render multiple 3D visual priors given arbitrary camera-object relations. OO: camera center; AA: object center; DD: camera-object distance.

Each input 3D asset is represented as a textured polygonal mesh A=(V,F,P)A = (V, F, P), where V={vi∈R3,i=1...N}V = \{v_i \in\mathbb{R}^{3}, i = 1...N\} denotes the mesh vertices, FF denotes the mesh faces, and PP contains appearance information such as materials and textures. We associate each asset with an object coordinate frame FO\mathcal{F}_O, whose origin is located at the object center A∈R3A \in\mathbb{R}^{3} (the red point in Figure 3), whose vertical axis specifies the canonical upright direction, and whose unit vector fO\mathbf{f}_O specifies the canonical object-facing direction. We further define a world coordinate frame FW\mathcal{F}_W and a camera coordinate frame FC\mathcal{F}_C. Let WTO∈SE(3)^{W}T_{O} \in SE(3) and WTC∈SE(3)^{W}T_{C} \in SE(3) denote the object-to-world and camera-to-world rigid transformations, respectively. The relative transformation from the object frame to the camera frame is therefore CTO=(WTC)−1WTO^{C}T_{O} = (^{W}T_{C})^{-1}{}^{W}T_{O}, which corresponds to the standard 6-DoF camera-object relative pose.

In our setting, we constrain the asset to remain upright, fix camera roll, and orient the camera optical axis toward the object center AA. Under these constraints, the camera-object relative pose can be compactly parameterized by β=(φ,θ,D)\beta= (\varphi, \theta, D), consisting of three components:

  • The object orientation angle φ∈[0,2π)\varphi\in[0, 2\pi), which represents the relative azimuth between the asset’s canonical facing direction and the camera-facing direction. Let fW\mathbf{f}_W denote the canonical facing direction transformed into the world coordinate frame, and let uW\mathbf{u}_W denote the world-up direction. For any vector x\mathbf{x}, define its projection onto the horizontal plane as Πh(x)=x−(x⊤uW)uW\Pi_h(\mathbf{x}) = \mathbf{x} - (\mathbf{x}^{\top}\mathbf{u}_W)\mathbf{u}_W, Let fh=Πh(fW)∥Πh(fW)∥2\mathbf{f}_h = \frac{\Pi_h(\mathbf{f}_W)}{\lVert\Pi_h(\mathbf{f}_W)\rVert_2}, qh=Πh(A−O)∥Πh(A−O)∥2\mathbf{q}_h = \frac{\Pi_h(A-O)}{\lVert\Pi_h(A-O)\rVert_2}, where OO is the camera aperture center. We define

φ=atan2⁡(uW⊤(qh×fh),qh⊤fh) mod 2π.\varphi= \operatorname{atan2}\left(\mathbf{u}_W^{\top}(\mathbf{q}_h \times\mathbf{f}_h), \mathbf{q}_h^{\top}\mathbf{f}_h\right) \bmod2\pi.

Thus, φ\varphi is the counterclockwise azimuth of the asset-facing direction with respect to the camera-facing direction OA→\overrightarrow{OA}.

  • The viewpoint angle θ∈[−π2,π2]\theta\in\left[-\frac{\pi}{2}, \frac{\pi}{2}\right], which represents the elevation of the camera with respect to the horizontal plane of the asset. Let r=O−A\mathbf{r} = O - A denote the vector from the asset center to the camera. We define

θ=atan2⁡(r⊤uW,∥Πh(r)∥2).\theta= \operatorname{atan2}\left(\mathbf{r}^{\top}\mathbf{u}_W, \lVert\Pi_h(\mathbf{r})\rVert_2\right).
  • The camera-object distance D>0D > 0, defined as the Euclidean distance between the camera center and the object center, measured in units of the object’s size,

D=∥O−A∥2s(A),D = \frac{\lVert O - A\rVert_2}{s(A)},

where s(A)s(A) is the largest extent of the asset’s axis-aligned bounding box. In practice each asset is rescaled to unit extent before rendering, so DD equals the camera distance in Blender units. Under our fixed 35mm focal length, D≈1D \approx1 corresponds to the object spanning the full image frame, and larger DD yields a smaller apparent object size.

Thus, β\beta should be interpreted as a three-parameter representation of the constrained camera-object relative pose, rather than as a general representation of arbitrary 3D geometry. Compared with [80], this explicit 3D camera-object parameterization allows us to systematically control object orientation, camera elevation, and camera-object distance during dataset generation.

Visual Prior Rendering. Given an arbitrary camera-object relation β\beta, the system renders a corresponding set of geometrically aligned 2D visual priors IβI_{\beta} (Figure 3) using a 3D renderer RR. We use a perspective camera with a default focal length of 35mm, while its extrinsic parameters are determined by β\beta. The asset remains upright and the camera is oriented toward its center. The rendered RGB images are further processed to provide Canny edges as additional visual priors.

Because all rendered modalities are produced from the same underlying 3D mesh and camera parameters, they are pixel-aligned and geometrically consistent with the specified camera-object relation β\beta. Images rendered under different values of β\beta therefore provide controlled variations in object orientation, camera viewpoint, and camera-shot distance while preserving exact 3D geometric supervision.

Diverse Image Description Generation

Our approach uses GPT-4o as a LLM to generate diverse text prompts for image descriptions (TimgT_{img}) for DM-based image generation; an example is in Figure 2. Our framework uses system prompting and few-shot prompting tricks which are broadly utilized for image captioning and QA composing [3, 43].

For the generation of TimgT_{img}, the LLM is guided to generate diverse descriptions of background context and object details given the object category cc. The diversity and rationality of the description directly influences the image generation quality (see Appendix § 13). More details are in Appendix § 8.

Controlled Image Generation

We utilize DMs to generate photorealistic images IsynI_{syn} given comprehensive visual prior IβI_{\beta}. A successful generation of IsynI_{syn} shall preserve ground truth camera-object relation information, and be of high fidelity. We first describe a traditional text-to-image DM pipeline, extend it to stacked multiple ControlNets using various images of IβI_{\beta}, and obtain precisely-controlled image generation.

Text-to-Image Generation. We use text-to-image DM (i.e., SDXL-1.0 [74]) as our backbone. The denoising U-Net, Gθ\mathcal{G}_{\theta} parameterized by θ\theta, conducts iterative denoising diffusion steps of Gaussian noise ϵ\epsilon for a total of TT time steps. A text-encoder transforms the text prompt TimgT_{img} into text embedding for guidance injection. The iterative denoising is defined as:

zt−1=Gθ(Timg,zt,t),t=T,…,1(1)z_{t-1} = \mathcal{G}_{\theta}(T_{img}, z_t, t), \quad t = T, \ldots, 1 \tag*{(1)}

where zT=ϵ∼N(0,I)z_T = \epsilon\sim\mathcal{N}(0, I), and z0z_0 is the latent variable of the generated image. It can be decoded to an RGB image by pretrained image decoder D\mathcal{D} as I0=D(z0)I_0 = \mathcal{D}(z_0).

Precise Controlling via Multiple ControlNets. ControlNet [79] guides image generation by injecting additional controlling features derived from various 2D image visual priors. There are dozens of off-the-shelf ControlNets, each pre-trained by a single type of visual prior. In our work, we leverage multiple ControlNets to capture comprehensive visual priors from IβI_{\beta} (depth, Canny edge, etc.) where each one is weighted by coefficients ww. The denoising process using multiple ControlNets (C\mathcal{C}) is defined as:

zt−1=Gθ(Timg,zt,t,∑k=1∣β∣wk⋅Ck(Iβk)),t=T,…,1.(2)z_{t-1} = \mathcal{G}_{\theta}\left(T_{img}, z_t, t, \sum_{k=1}^{|\beta|} w_k \cdot\mathcal{C}^{k}\left(I_{\beta}^{k}\right)\right), \quad t = T, \ldots, 1. \tag*{(2)}

Similarly, the output generated image Isyn=D(z0)I_{syn} = \mathcal{D}(z_0) and IsynI_{syn} is expected to have ground truth camera-object relations.

Text Instruction Generation

We utilize GPT-4o as a LLM to generate QA pairs on camera-object relation (TqaT_{qa}) for visual instruction tuning (more details in Appendix § 9). Specifically, the LLM is prompted to generate QA pairs given the camera-object relation β\beta. We subdivide each component of β\beta into categorical types for effective referring through textual response by MLLMs:

  • Object orientation angle φ\varphi is classified into eight object orientation types, each covers a bin of π4\frac{\pi}{4} azimuth (i.e. ”right”, ”front right”, ”front”, ”front left”, ”left”, ”back left”, ”back”, ”back right”).

  • The viewpoint angle θ\theta is classified into three viewpoint types, each covering a range of π3\frac{\pi}{3} elevation angle (i.e. ”horizontal”, ”top”, ”bottom”).

  • The camera-object distance DD (§ 3.1) is measured in multiples of the object’s size, with D≈1D \approx1 corresponding to the object filling the image frame. The three camera-shot types (i.e. ”close-up”, ”medium-shot”, ”long-shot”) are decided by two cutoff values, 1.25 and 3.0.

Examples of generated multiple-choice style QA pairs as MMVP benchmark [21] are showed in Figure 2. For question generation, we prompt LLMs to generate template questions for each prediction task of object orientation, viewpoint type, and camera-shot type. We list all possible options, including the ground truth β\beta, in random order. MLLMs are expected to choose the option according to β\beta.

Experiments

Visual Instruction Generation

Data Source. Our framework uses open-sourced 3D assets datasets from ObjaverseXL [81, 82] and ShapeNet [83]. Partial 3D assets are labeled following the synset list from ImageNet-1K [84]. We leverage this subset and define the asset category by its synset label. In particular, we manually select 100 synsets with explicit ”facing” orientation from the original list of ImageNet. In total, there are 1196 assets used for synthetic visual instruction generation.

ControlNets. We utilize SDXL-1.0 [74] as the backbone of ControlNets. For selection of multiple visual priors, we found using both Canny edges and relative depth provides the best mix of camera-object relation preservation and image quality. The diffusion step T=30T = 30, and relative weights for depth and Canny edge ControlNets are 0.5 and 0.8, respectively. Several seconds are needed to generate an image by a single H100 GPU. Ablations and user studies are shown in § 4.6 and § 4.7.

Ultimate3D Dataset and Benchmark.

Our framework creates the Ultimate3D, a large-scale 3D-grounded 2D visual instruction dataset whose samples are generated from explicit 3D assets and controlled camera geometry, providing precise ground-truth camera-object relation annotations. Ultimate3D consists of 85K synthetic images and 18K cropped images of a single human per photograph and corresponding human orientation labels from MEBOW [85]. For the synthetic images, we use 1180 3D assets covering 100 categories. For each 3D asset, there are 72 possible camera-object relations (8 orientations, 3 viewpoints, and 3 camera shot types) and we render a corresponding synthetic image for each relation. Then, each synthetic image is described by up to 3 textual QAs corresponding to object orientation, camera viewpoint, and camera-shot type, yielding a total of 240K corresponding QAs. Examples of synthetic images are in Appendix § 16. The images from MEBOW are useful because otherwise there is a strong lack of 3D assets of human which we find crucial to fine-tuning. The MEBOW images only have one QA corresponding to human orientation.

The Ultimate3D benchmark (Figure 4) has two sets: (1) Synthetic Set. It contains 1200 synthetic images and 3600 QAs generated by our framework. Each image and QA has been manually reviewed to ensure correctness. This set covers all 100 categories with explicit facing orientation. The 3D assets utilized for benchmark generation do not overlap with those used for dataset generation. (2) Real Set. It contains two portions. The first portion is 2443 real images collected from Pascal3D+ [86] covering 10 categories (e.g. aeroplane, bicycle, etc.). Each image corresponds to a QA based on provided labels of object orientation or camera viewpoint. The second portion is 800 real images of humans collected from the MEBOW dataset [85]. Each image corresponds to a QA on human orientation based on provided labels. In total, there are 3243 VQAs in the Real set. We resample the real images to keep a uniform distribution for each orientation and viewpoint type.

Qualitative results showing model responses on synthetic Ultimate3D and real Pascal3D+ and MEBOW images

Figure 4. Qualitative Results. We show responses by GPT-5.5, Gemini-3.5-Flash with high thinking, Gemma-4-31B-it, and finetuned Gemma-4-31B-it on the Ultimate3D benchmark. Each model is asked questions (in gray boxes) regarding object orientation, camera viewpoint, and camera-shot type, together with the input images on the left. The model responses illustrate that the Gemma-4-31B-it model finetuned on Ultimate3D can precisely recognize camera-object relations on synthetic images. GPT-5.5 and Gemini-3.5-Flash with high thinking sometimes provide correct answers, but also exhibit inconsistent predictions across relation types. The original Gemma-4-31B-it frequently fails to produce the correct camera-object relation labels, while the finetuned model provides substantially more accurate and task-relevant answers.

Since no camera-shot information is provided in the real image set, the QAs in the real set exclude camera-shot type questions.

Both the Ultimate3D dataset and the benchmark have no camera-object relation bias, which is very uncommon in the real image datasets (e.g. most images capture objects facing straight to the camera [86]). In contrast, in our dataset the number of images in each orientation, viewpoint, and camera-shot type are evenly distributed.

Advancing 3D-Aware MLLMs

We explore how much our synthetic Ultimate3D dataset can improve the MLLMs recognition capability on camera-object relation. The evaluation results on both synthetic and real image VQA benchmarks indicate the fine-tuned baseline models are significantly improved from random guess and outperform commercial SOTAs.

Training Details. We fine-tune and evaluate several recent open-source MLLM families, including LLaVA-1.5/1.6 [4], Llama-3.2-Vision-11B [9], Gemma-4 models [10], and Qwen-based models [11, 12]. All models are fine-tuned for one epoch on the Ultimate3D training set. To evaluate the effect of training on Ultimate3D, we compare each base model with its corresponding fine-tuned version. We adopt two fine-tuning protocols depending on the model family. For LLaVA and Llama-3.2-Vision, we follow their official full fine-tuning recipes [4, 9], updating the MLP vision-language connector and the LLM decoder while freezing the vision encoder. For LLaVA, we use BF16 precision, a learning rate of 2×10−52 \times10^{-5}, cosine scheduling with a warm-up ratio of 0.03, zero weight decay, and gradient checkpointing. For Llama-3.2-Vision-11B, we use BF16 precision, fused AdamW, a learning rate of 2×10−52 \times10^{-5}, a per-device batch size of 2 with gradient accumulation of 4, and gradient clipping at 1.0. For Gemma-4 and Qwen-based models, we use parameter-efficient fine-tuning primarily based on LoRA. For the Qwen3.5 and Qwen3-VL series, we use Unsloth-based parameter-efficient fine-tuning with a hybrid trainable-parameter policy: the vision encoder is kept frozen, the language-model attention and MLP projections are adapted with LoRA (rank 16, α\alpha=16, dropout 0, no bias terms, no RSLoRA), and the vision–language connector (merger) is optimized as full weights.

(merger) is optimized as full weights. For Gemma-4 models, we use plain LoRA with rank 16, α\alpha = 32, and dropout 0.05, applied to compatible linear layers in the language model and vision embedding projection. Gemma-4 models are trained in BF16 using fused AdamW, an effective batch size of 4, and a constant learning rate of 1×10−41 \times10^{-4} without warmup or weight decay. For Gemma-4-31B, we resumed from the best intermediate checkpoint with a learning rate of 3×10−53 \times10^{-5}, cosine decay, and weight decay of 0.01 due to late-stage instability. Training takes approximately 12 hours on 4 H100 GPUs for LLaVA and Llama-3.2-Vision, up to 2.5 days on a single H100 GPU for Gemma-4, and up to 8 days on a single RTX 6000 GPU for Qwen-based models.

Evaluation Settings. We evaluate the baseline MLLMs and their fine-tuned versions on Ultimate3D benchmark and MMVP benchmark [21]. MMVP benchmark is an intentionally selected real image dataset which is claimed to be difficult distinguished by MLLMs. We manually label 89 VQAs regarding camera-object relations. During evaluation, MLLMs’ textual responses are evaluated by GPT-4o, which is prompted to judge if the responses correspond to the ground truth answers. We calculate the accuracy rate (%) of the MLLMs responses as the evaluation metric. We also include commercial SOTA models (e.g. GPT-4o and Claude-3.5-Sonnet) in the evaluation.

Quantitative Results. The quantitative results of response accuracy (%) are shown in Table 1. We assess the effect of training on Ultimate3D through base-to-fine-tuned comparisons within each model family. Overall, we observe a consistent trend: off-the-shelf MLLMs struggle with camera-object relation recognition, whereas task-specific fine-tuning on Ultimate3D substantially improves performance across multiple model families. This difficulty appears even for strong general-purpose vision-language models, including LLaVA, Llama-3.2-Vision, Gemma-4, Qwen-based models, and closed commercial MLLMs. Several baseline models tend to collapse to a small set of frequent answers, such as predicting “front” orientation or “top” viewpoint regardless of the actual image content. As a result, their accuracies are often close to random-guess levels, e.g., 33.3% for three-way viewpoint recognition and 12.5% for eight-way orientation recognition. This weakness may be related to biases in commonly used visual

Table 1: Quantitative Comparisons. We evaluate open-source and closed-source MLLMs on the Ultimate3D and MMVP benchmarks. LLaVA and Llama models are fine-tuned using their original full fine-tuning recipes, while Gemma-4 and Qwen models are fine-tuned with LoRA-based parameter-efficient fine-tuning (for Qwen, the vision-language merger is also fully trained). Results report MLLM response accuracy (%) for object orientation, camera viewpoint, and camera-shot recognition. Superscripts indicate the fine-tuning setup: F full fine-tuning and L LoRA fine-tuning.

MLLMsOrientation (% ↑)Viewpoint (% ↑)Camera-Shots (% ↑)
Ultimate3DMMVPUltimate3DMMVPUltimate3DMMVP
BothSynRealBothSynRealSyn
LLaVA-1.5-7B18.311.122.525.031.838.425.329.246.066.7
LLaVA-1.6-7B18.214.720.325.036.843.929.833.346.671.4
LLaVA-1.5-13B17.918.717.422.738.935.042.841.744.661.0
LLaVA-1.6-13B16.116.416.015.930.730.331.133.342.361.9
Llama-3.2-V-11B5.65.35.76.819.826.213.638.129.625.0
Finetuned LLaVA-1.5-7BF^{\mathrm{F}}68.885.658.854.570.681.360.066.794.066.7
Finetuned LLaVA-1.6-7BF^{\mathrm{F}}71.586.962.461.471.383.359.566.794.150.0
Finetuned LLaVA-1.5-13BF^{\mathrm{F}}70.085.660.850.069.781.957.766.794.276.2
Finetuned LLaVA-1.6-13BF^{\mathrm{F}}72.488.163.165.972.383.061.875.094.871.4
Finetuned Llama-3.2-V-11BF^{\mathrm{F}}74.287.366.450.069.179.658.758.393.176.2
Qwen3.5-2B28.036.120.234.140.637.144.150.045.242.9
Qwen3.5-4B37.042.831.345.545.764.327.350.045.547.6
Qwen3.5-9B39.648.331.140.947.163.730.850.037.728.6
Qwen3-VL-4B21.928.615.345.545.752.139.433.343.447.6
Qwen3-VL-8B34.341.926.952.347.159.734.645.844.938.1
Qwen3-VL-32B41.146.735.656.852.566.438.850.044.357.1
Finetuned Qwen3.5-2BL^{\mathrm{L}}68.080.460.261.466.876.557.162.591.770.7
Finetuned Qwen3.5-4BL^{\mathrm{L}}71.379.063.560.971.079.063.066.792.172.8
Finetuned Qwen3.5-9BL^{\mathrm{L}}73.382.564.063.671.873.570.262.891.971.2
Finetuned Qwen3-VL-4BL^{\mathrm{L}}67.583.856.268.272.881.559.575.191.573.3
Finetuned Qwen3-VL-8BL^{\mathrm{L}}69.580.862.562.771.276.267.170.889.777.0
Finetuned Qwen3-VL-32BL^{\mathrm{L}}73.183.565.569.573.482.365.173.392.278.7
Gemma-4-E2B-it25.628.224.040.939.548.230.937.537.747.6
Gemma-4-E4B-it29.835.226.638.644.058.030.158.339.228.6
Gemma-4-26B-A4B-it47.354.643.056.852.664.241.166.736.947.6
Gemma-4-31B-it44.955.238.952.352.863.742.154.240.152.4
Finetuned Gemma-4-E2B-itL^{\mathrm{L}}67.380.059.861.466.776.856.762.591.871.4
Finetuned Gemma-4-E4B-itL^{\mathrm{L}}65.778.658.259.169.476.262.858.389.876.2
Finetuned Gemma-4-26B-A4B-itL^{\mathrm{L}}71.983.265.361.467.776.858.658.392.471.4
Finetuned Gemma-4-31B-itL^{\mathrm{L}}73.986.866.345.569.880.159.762.592.276.2
GPT-4o43.151.038.540.954.163.544.970.841.742.9
GPT-4o-mini43.552.338.338.654.563.945.166.741.542.9
Claude-3-Sonnet43.152.637.536.454.364.244.566.741.747.6
Claude-3.5-Sonnet40.347.735.854.555.665.945.366.741.842.9
Gemini-3.5-Flash (min. thinking)56.866.950.861.461.468.454.483.337.152.4
Gemini-3.5-Flash (high thinking)65.373.860.356.861.667.056.379.233.842.9
Claude-Sonnet-4.636.444.128.950.056.467.145.866.740.947.6
GPT-5.554.162.545.950.559.269.748.958.337.352.4
Claude-Opus-4.746.052.040.445.557.267.147.562.544.352.4

Table 1.

instruction datasets, where most images are captured from canonical front-facing directions and from common side or above viewpoints. The poor off-the-shelf performance across both older and more recent MLLM families suggests that camera-object relation understanding remains underrepresented in existing MLLM training data.

Fine-tuning on our synthetic Ultimate3D dataset substantially improves camera-object relation recognition across all evaluated open-source model families. On the Ultimate3D benchmark, fine-tuned LLaVA models significantly outperform their corresponding base models across object orientation, camera viewpoint, and camera-shot recognition. For example, fine-tuned LLaVA-1.6-13B improves orientation accuracy from 16.1% to 72.4%, viewpoint accuracy from 30.7% to 72.3%, and camera-shot accuracy from 42.3% to 94.8%. The smaller LLaVA-1.6-7B model achieves similarly competitive performance, reaching 71.5%, 71.3%, and 94.1% on these three tasks, respectively. This suggests that the ability to distinguish camera-object relations does not necessarily require a very large backbone, but instead benefits strongly from task-specific supervision.

The same conclusion holds for Llama-3.2-Vision, Gemma-4, and Qwen-based models. Llama-3.2-Vision-11B obtains the largest average improvement among the models in Table 1, with an average gain of 60.46 percentage points across the three camera-object relation tasks. The original released model often refuses to answer some recognition questions, e.g., responding that it cannot provide information about the individual in the image, while the fine-tuned model produces valid relation predictions and achieves the best orientation accuracy in Table 1. These results indicate that Ultimate3D supervision can correct both answer-format failures and visual relation recognition failures.

Recent Gemma-4 and Qwen-based models further confirm that the benefit of Ultimate3D supervision is not specific to the LLaVA architecture. Before task-specific fine-tuning, Gemma-4 models achieve only 36.0–51.1% overall accuracy when averaging all reported relation-recognition columns. After LoRA fine-tuning, they reach 69.4–71.3% overall accuracy, corresponding to an average improvement of 26.3 percentage points. The improvements are consistent across all three relation types, with average gains of 20.0–37.5 percentage points on object-direction recognition, 9.2–26.7 percentage points on viewpoint recognition, and 38.0–49.1 percentage points on camera-shot recognition.

Qwen-based models show a similar pattern. Their base versions achieve 37.3–48.9% overall accuracy, while the fine-tuned versions reach 69.5–75.7% overall accuracy, improving by 30.5 percentage points on average. In particular, fine-tuned Qwen3-VL-32B achieves the strongest overall performance among the evaluated open models in Table 1, reaching 75.7% average accuracy across all reported columns. It also obtains strong performance on both synthetic and real subsets, suggesting that the learned camera-object relation knowledge is not restricted to rendered synthetic images.

Among closed commercial MLLMs, newer models generally perform better than earlier GPT-4o- and Claude-3-era models on some orientation and viewpoint columns, but the improvement is not uniform across tasks. For example, Gemini-3.5-Flash with the high thinking setting achieves stronger orientation accuracy than its minimal-thinking counterpart, improving from 56.8% to 65.3% on the Ultimate3D orientation column. However, increasing the thinking budget does not consistently improve all relation types: its camera-shot accuracy decreases from 37.1% to 33.8%. This suggests that additional reasoning budget can help in some spatial-relation cases, but it does not resolve the lack of grounded camera-object relation understanding.

Compared with closed commercial MLLMs, our fine-tuned open models achieve especially strong performance on the Ultimate3D relation-recognition tasks. Although recent commercial models improve over earlier commercial baselines on some orientation and viewpoint columns, they remain substantially weaker on several Ultimate3D tasks, especially camera-shot recognition, where fine-tuned open models consistently reach around 90% accuracy or higher. Overall, these results demonstrate that Ultimate3D provides effective supervision for camera-object relation understanding, improves both synthetic and real-image evaluation performance, and generalizes across multiple MLLM families, including LLaVA, Llama, Gemma, and Qwen.

Qualitative Results.

We show qualitative comparisons in Figure 4. Commercial SOTA models (e.g. Gemini-3.5-Flash) and open-source models (e.g. Gemma-4-31B-it) struggle with precise recognition of all three types of camera-object relations. For the two commercial models, their object-orientation predictions are often unreliable despite producing plausible natural-language explanations. We also observe systematic biases in viewpoint and camera-shot recognition: both commercial models tend to underestimate viewpoint angle, e.g., mistaking top views as horizontal, and overestimate camera-shot distance, e.g., mistaking medium-shot images as long-shot. The original Gemma-4-31B-it model also shows strong response biases across all types of questions. Its orientation predictions are biased mostly toward “Left” and “Right”, its viewpoint predictions are dominated by “Horizontal”, and its camera-shot predictions are biased toward “Long-shot”. In comparison, the Gemma-4-31B-it model fine-tuned on our Ultimate3D dataset provides more plausible predictions across all three tasks on both the synthetic and real subsets of our benchmark. Additional comparisons are provided in Supplemental Sec. H.

We hypothesize that these behavioral biases arise from dataset bias and the lack of camera-object visual instruction data in current MLLM training pipelines. Without modifying the model architecture or training scheme, our fine-tuned Gemma-4-31B-it model substantially improves camera-object relation recognition and outperforms strong commercial models on several Ultimate3D tasks, despite relying only on task-specific supervision. This indicates that the lack of targeted camera-object relation data is a major factor behind current MLLMs’ weakness on camera-object relation recognition.

Limitations. The numerical prediction of camera-object relation β\beta is more challenging for MLLMs compared to the categorical prediction we report. Preliminary tests show MLLMs can handle numerical prediction of camera-object relation for a single category, but struggle with a hundred categories as in our dataset. Since our main contribution is the discovery of the dataset bottleneck and the generation pipeline, we leave addressing this limitation as future work.

Generalization of Ultimate3D to General VQA

To explore the generalization of 3D camera-object relation capturing with general VQA tasks, we fine-tune LLaVA-1.6-13B models by a half-half mixture of Ultimate3D (240K VQAs) and randomly sampled 240K VQAs from LLaVA-Instruct-665K [4]. Table 2 presents the dataset mixture finetuning performance on VQAv2 [48], our Ultimate3D, and MMVP benchmarks. The results demonstrate that incorporating Ultimate3D significantly enhances camera-object relation reasoning while maintaining general VQA performance.

MLLMsVQAv2 [48]Ultimate3D^{*}MMVP^{*}
LLaVA-1.6-13B [4]80.0029.7037.03
Ulti3D (10%)78.7779.8370.77
Ulti3D (50%) + (50%) [4]80.0174.7265.80
GPT-4o/46.3051.57
Claude-3.5-Sonnet/45.9054.70

Table 2. Generalization. We finetune LLaVA-1.6-13B model using a mixture of both general VQA and camera-object relation VQA datasets. Without degradation on general VQA tasks, fine-tuned model improves its ability on camera-object relation recognition and beats SOTA (i.e. GPT-4o) by 28.42% accuracy on Ultimate3D benchmark. * denotes the average accuracy (%) across all 3 types of camera-object relation VQAs in Table 1.

Ablations on Image Generation Pipeline

In Figure 5, we ablate our image generation pipeline and illustrate better quality than [80] by: (1) removing the visual priors; and (2) using alternative image generation backbones; (3) changing the group of visual priors given to the image generator. Compared to text-only SDXL and [80], we find that multiple visual priors may significantly improve the robustness of image generation under complex camera-object relation (especially for viewpoints of the object’s back-side, and for 3D assets with low level-of-detail). In particular, the depth and Canny edge prior are crucial. However, we find that adding more priors (e.g., RGB, semantic masks, normal maps) does not explicitly improve quality. Moreover, our SDXL backbone

Ablations on Image Generation Pipeline

Figure 5. Ablations on Image Generation Pipeline. Our framework shows superior performance by using multi-ControlNet with SDXL backbones, and introducing both Canny edges and depth maps as visual priors. It consistently delivers better robustness and quality compared to text-only SDXL and [80]. When these visual priors are removed (top-right: removing all visual priors; bottom-right: removing depth prior), the generator often fails to produce good results and brings lower generation success rate (see § 4.7, [80] only achieves 55.11% success rate v.s. ours 93.07%) outperforms other alternatives (i.e. SD-V1.5 [73]). Discussions in Appendices § 12 and § 13.

User Study on Dataset Quality

We investigate using text-image aligning metric and pose-estimation model prediction to evaluate quality of our Ultimate3D dataset but find those metrics to be deficient (Appendix § 11). Instead, we perform a comprehensive user study as a quality check of Ultimate3D dataset. As Appendix § 14 shows, the user will see two images side-by-side (Figure G4): an RGB image rendered by Blender (representing IβI_{\beta}), and a synthetic image generated by our method using IβI_{\beta}. Preserving the RGB image’s 3D geometry and structure in the synthetic image is regarded as success. Our study includes 225 randomly sampled image pairs where each image is independently reviewed by 3 users.

User study UI page showing a conditioning image of shape and structure alongside a generated image

Figure G4. User Study UI Page. Users will see a pair of side-by-side images. The left hand side is an RGB image generated by Blender, and the right hand side is the synthetic image generated by DMs using that RGB image as guidance prior. A vivid preserving of 3D geometry and structure in the synthetic image is a success. Users are required to provide a binary answer regarding the success of the image generation.

Overall success rate is 93.07% (agreement rate among 3 users is 94.4%). In comparison, we perform another similar-objective user study on [80]. The overall success of this alternative method was only 55.11% (we explain the cause of this reduced performance in § 4.6).

Conclusion

In this work, we discover the dataset bottleneck of MLLMs for distinguishing camera-object relations. Given only 3D assets, our framework is able to generate unlimited image-text pairs of diverse camera-object relations with no distribution bias. We also provided Ultimate3D dataset and benchmark MLLM community. Moreover, fine-tuned open-sourced MLLMs using our dataset outperform SOTA commercial models by a significant margin. We plan to extend our framework for multi-object image generation with additional viewpoints, and improve numerical prediction capability on camera-object relation.

Acknowledgments

This work was performed during Liu He’s internship at Amazon, and later extended by Ruiqi Xu and Baris Tezcan at Purdue University. Ruiqi Xu fine-tuned Qwen3 series and evaluated GPT5.5 & Claude models, and Baris Tezcan fine-tuned Gemma-4 series and evaluated Gemini models. We thank Jingjing Zheng, Tony Qi, Nan Qiao, Ke Zhang, and Yuyin Sun for their insightful discussions and valuable contributions to the formulation of the ideas in this work.

Appendix Overview

Below is a summary of the contents in each section of this supplemental material:

  • Sec. A: Details of text prompt generation as image description utilized for controlled image generation.

  • Sec. B: Details of VQA prompt generation given the ground truth camera-object relations.

  • Sec. C: Details of LLM-based grading for MLLM response evaluation.

  • Sec. D: Preliminary studies on data curating for camera-object relation VQA dataset by text-image aligning metrics, and by pose estimation models.

  • Sec. E: Ablations on selections of DM backbones.

  • Sec. F: Factors influence generation quality of our image generation pipelines.

  • Sec. G: Details of user study on image quality evaluation.

  • Sec. H: More qualitative comparisons between finetuned LLaVA and Gemma models to commerical SOTA models.

  • Sec. I: 48 image examples illustrate the diversity of Ultimate3D dataset and benchmarks on object categories, camera-object relations, and background context.

  • Sec. J: Generalization to real images: evaluation of base and Ultimate3D-fine-tuned models on the independently collected ImageNet3D benchmark.

  • Sec. K: Failure analysis of fine-tuned models: residual weak relations, error structure, boundary effects, and annotated image conditions.

  • Sec. L: Additional discussions and insights on using synthetic dataset generated by our framework for MLLM finetuning.

  • Sec. M: Algorithm of our 3D visual instruction dataset generation pipeline.

Image Description Generation

We provide system prompts given to GPT-4oto generate context description for image generation. Corresponding section in main paper is § 3.2. In the prompt, we provide one-shot example for better robustness of the text generation.

Specifically, generated text prompt will be merged with default positive prompt like ”detailed, 4K, 35mm photograph, professional”, and customized camera-object prompt ”the image shows a [β] view of a [c]”, in order to enhance image generation quality.

SYSTEM_PROMPT_FOR_IMAGE_GEN = "You are an expert in
generating concise and diverse descriptions of an
object to guide image generation model like DAll-E3.
You should use your common sense to generate 1
sentence description of the given object.
Additionally, you should also generate 1 sentence of
a common real-world scene given the object. Both
sentences of descriptions should not include more
than one of the given object to avoid ambiguity.

Example Input: Please generate the visual prompt of
chicken.
Example Output: A white and brown chicken standing in
a cage made of metal bars. The chicken has a long,
curved beak and its feathers are fluffy and white. "

Text Instruction Generation

We provide the system prompt to generate QA pairs given camera-object relation as below. Corresponding section in main paper is § 3.4. In the prompt, we provide few-shot example for better instructions for diversity of the generated text.

SYSTEM_PROMPT_FOR_VQA_GEN = "You are an AI visual
assistant, and you are seeing a single image. What
you see are provided with several sentences,
describing the same image you are looking at. Answer
all questions as you are seeing the image.

Design a conversation between you and a person asking
about this photo. The answers should be in a tone
that a visual AI assistant is seeing the image and
answering the question.
Ask diverse questions and give corresponding answers.

Include questions asking about the visual content of
the image, including the object orientation and its
corresponding azimuth degree, the camera viewpoint
of the image and corresponding camera elevation
angle, the camera shot type of the image and the
distance from camera to the target object. Only
include questions that have definite answers:
(1) one can see the content and its orientation degree
in the image that the question asks about and can
answer confidently;
(2) one can recognize camera viewpoint and camera shot
of the image that the question asks about and can
answer confidently;
The question can be multiple choice questions given
the choice, or open-ended questions. For multiple
choice, the option expression can be diverse but
with the same meaning. For example, "front" also
means "directly towards the camera"; "back" also
means ’away from the camera’, etc. Use the common
sense to make options diverse if possible.
Provide at most 5 pairs of question and answer pairs.
Prior the confident questions. Do not ask any
question that cannot be answered confidently.

You should ask questions from multiple QA templates as
the below examples show.

Example Input Description:
The image shows a front view of police van. A blue and
white police van with the word "POLICE" emblazoned
on the side is parked on a city street. Nearby, a
couple of officers are discussing a recent incident
while pedestrians walk by, some glancing curiously
at the vehicle.

The police van is with an azimuth of 181.518 degree, facing Front direction. The elevation angle of the camera to the police van is 81.41 degree, the camera viewpoint is Horizontal view. The relative distance from the camera to the police van is 1.132 meters, the camera shot type is Close-up.

Example Output Question and Answer Pairs:
Question:
From the camera’s perspective, is the police van in the picture facing straight or oriented at an angle?
  Options: (a) Directly towards the camera (b) At an angle
===
Answer:
(a) Directly towards the camera
===
Question:
Is the police van in the picture facing the camera or away from the camera? Options: (a) Away from the camera (b) Facing the camera
===
Answer:
(b) Facing the camera
===
Question:
Which direction is the police van facing in the image?
  Options: (1) Back (2) Front
===
Answer:
(2) Front
===
Question:
Is the police van facing back or front from the camera’s perspective? Options: (a) Back (b) Front
===
Answer:
(b) Front
===
Question:
Is the photo taken directly above the police van or from the side? Options: (a) Taken directly (b) From the side
===
Answer:
(b) From the side
===
Question:
Is the photo taken far away the police van or taken closely?
===
Answer:
The relative distance from the camera to the police van is 1.132 meters, thus it is with a close-up shot. This indicates the photo is taken closely.
===
Question:
What is the elevation viewpoint of the image? Options: (1) Top (2) Horizontal (3) Bottom
===
Answer:
(2) Horizontal
===
Question:
Which camera shots is the image? Options: (1) Close-up (2) Medium-shot (3) Long-shot
===
Answer:
(1) Close-up"

LLM-based Response Grading

We provide the system prompt for evaluation of the MLLM response to our camera-object relation multiple choice questions.

SYSTEM_PROMPT_FOR_GRADING_MLLM_RESPONSE =’You are a helpful and precise assistant for checking the quality of the answer. You should review all listed choices and compared to the Answer content, and judge whether the Answer is correct (yes), or not (no). You should only focus on the major content of the answer, not the detailed number or symbol. Please answer in only yes or no’

Metric Limitations for Evaluating Camera-Object Relation

In Figure D1, we show an example of using ImageReward [87] for the dataset curation by evaluating the alignment between generated image and text prompts. Preliminary test indicates that text-image aligning metric is not sensitive to camera-object relation text prompts. Thus we may not rely on those metrics for data curation purpose.

Four synthetic snowmobile images comparing text-image alignment scores

Figure D1. Limitations of Text-Image Aligning Metrics. We evaluate the ImageReward [87] (IR) between the text prompt regarding object orientation with above 4 synthetic images. The image matches the correct camera-object relation text prompt (”front”) may reach the highest score. However, image with the highest matching score (it faces to ”left” direction) is not the correct one with expected orientation. This failure is prevailing across all categories.

Alternatively, in Figure D2, enlighten by [80], we train a transformer-based pose estimation model on a 10-fold manner for generated images on each single category. The images with less prediction errors may indicate successful generation. However, the pose estimation model is not robust to complex objects and diverse camera-object relations in our dataset. Using pose estimation model prediction error is not robust for data curation purpose.

Ground-truth and generated images illustrating failures of pose estimation

Figure D2. Failures of Pose Estimation Model. Enlighten by [80], we train a transformer-based pose estimation model on a 10-fold manner for generated images on each single category. The images with less prediction errors may indicate successful generation. However, the pose estimation model is not robust to complex objects and diverse camera-object relations in our dataset.

Ablation of DM Backbones

In Table E1, we perform quantitative comparisons on general image visual quality between different DM backbones: SD V1.5 and SDXL. We randomly sample 1,000 generated images (IsynI_{syn}), each image corresponds to: (1) the RGB visual prior rendered using Blender (one element of IβI_{\beta}); (2) the text prompts (TimgT_{img}) provided to the diffusion model

AblationsCLIP-T ↑CLIP-I ↑DINO ↑IR ↑FID ↓
Ours (SD-V1.5)29.8083.5669.36-0.29289.99
Ours (SDXL)34.4087.4470.790.23256.94

Table E1. Ablations on DM Backbones. SDXL provides better fidelity (CLIP-I, DINO), realism (FID), and text-image alignment (CLIP-T, IR) than other alternatives.

for image generation. We evaluate the generated images on the following aspects:

  • Prompt following. CLIP-T (CLIP-Text Score [88]) and IR (Image Reward [87]) metrics are employed to assess the alignment between the generated images and the input prompts. The results indicate that images produced by the SDXL backbone exhibit superior prompt alignment.

  • Fidelity. CLIP-I (CLIP-Image Score [88]) and DINO Score [14] are used to measure the similarity between the visual prior images and the synthesized outputs. Findings demonstrate that SDXL more effectively preserves fidelity to the visual prior.

  • Realism. FID [89] is used to compare the distributions of generated images with a subset of LAION-aesthetic [51] images, revealing that SDXL significantly surpasses SD V1.5 in terms of image quality and realism.

Factors influencing Image Generation

Several factors may lead to unreasonable artifacts in our image generation pipeline. In Figure F3, the upper-left example shows that bottom-view camera-object relations can introduce physical anomalies, such as the tricycle appearing with its front lifted unnaturally. The upper-right example highlights that low-quality 3D assets can produce fragmented depth maps or Canny edge visual priors, resulting in unrealistic generated images. The lower-left example shows that small subjects viewed from a long-shot perspective may lead to duplicate subjects in the generated image. The lower-right example illustrates how complex 3D asset structures can cause texture blending issues, particularly where intricate geometry is present.

Factors influencing Image Generation

Figure F3. Factors influencing Image Generation. The upper-left example shows that bottom-view camera-object relations can introduce physical anomalies, such as the tricycle appearing with its front lifted unnaturally. The upper-right example highlights that low-quality 3D assets can produce fragmented depth maps or Canny edge visual priors, resulting in unrealistic generated images. The lower-left example shows that small subjects viewed from a long-shot perspective may lead to duplicate subjects in the generated image. The lower-right example illustrates how complex 3D asset structures can cause texture blending issues, particularly where intricate geometry is present.

There are several ways to reduce the influences from those factors. First, choosing 3D assets with reasonable number of meshes, and with plausible textures may filter the low-quality and texture blending failures. Second, using simple and compact image description prompts to generate long-shot camera-shot images will significantly reduce the duplicate subject failures. Reduce the image generation with high elevation angle from bottom view will significantly reduce physical anomaly.

User Study Details

The Figure G4 shows the UI page of our user study. We randomly selected 200 images from the Ultimate3D dataset. Each image corresponds to an RGB prior generated by Blender. During the user study, users will see a pair of side-by-side images. One is an RGB image generated by Blender, and the other is the synthetic image generated by DMs using that RGB image as guidance prior. A vivid preserving of 3D geometry and structure in the synthetic image is a success. Users are required to provide a binary answer regarding the success of the image generation. Each image pair will be manually reviewed by at least 3 users.

Additional Qualitative Results

Based on Ultimate3D benchmark, we provide additional Figure H6 to show more qualitative comparisons between finetuned LLaVA model to commercial SOTAs. The finetuned LLaVA model outperforms other models.

Additional Qualitative Results with Gemma

Figure H6. Additional Qualitative Results with Gemma. We evaluate camera-object relation recognition for GPT-5.5, Gemini-3.5-Flash under the high thinking setting, Gemma-4-31B-it, and fine-tuned Gemma-4-31B-it on the Ultimate3D benchmark. Each model is asked questions, shown in gray boxes, about object orientation, camera viewpoint, and camera-shot type, together with the input images on the left. The responses show that fine-tuning Gemma-4-31B-it on Ultimate3D substantially improves camera-object relation recognition compared with the base Gemma-4-31B-it and closed-source models.

Additional Visuals of Ultimate3D Dataset

In Figure L9, we shows more examples of diversity on object categories, camera-object relation, and background contexts.

Diversity of Ultimate3D across camera-object relation settings

Figure L9. Diversity of Ultimate3D. Our Ultimate3D dataset and benchmark cover 100 categories of objects, range diverse camera-object relation settings, and provide plausible image quality. (Each row shows images with the same orientation but in diverse subject and context.)

Generalization to Real Images: ImageNet3D

To test whether the camera–object relation knowledge learned from our synthetic data transfers to real photographs, we evaluate on ImageNet3D [90], a benchmark of ∼69\sim69K ImageNet images with human-annotated 3D object poses across 189 categories, collected independently of our pipeline. We convert its validation split into our VQA format, keeping images that contain a single quality-checked annotated object (23,224 images, 186 categories). Annotated azimuth and elevation are discretized into our eight orientation and three viewpoint classes; we validated this mapping against PASCAL3D+ labels on the 1,109 images shared by both datasets (97.5% and 99.0% agreement, respectively). As ImageNet3D does not annotate camera shot, we derive shot labels from object framing (the maximum bounding-box extent relative to the image), calibrated to our three shot classes, with ambiguous boundary ranges excluded. Since real photographs are strongly biased toward frontal, eye-level views, we sample an answer-balanced evaluation set of 1,200 questions per task. ImageNet3D is used for evaluation only.

Table J2 compares each baseline with its Ultimate3D-fine-tuned counterpart. Fine-tuning improves every model on every task (+15.2, +7.8, and +7.8 points on average for orientation, viewpoint, and camera shot). The baselines largely collapse onto frequent answers (e.g., “Front” for most orientation questions), whereas the fine-tuned models spread predictions across all classes and gain most on rear and oblique views, indicating improved camera–object understanding rather than better answer priors. The one residual gap is the Long-shot class, whose accuracy decreases after fine-tuning; we attribute this to our synthetic long shots being clean, object-centered renders, unlike cluttered real-world wide shots, and leave closing this gap to future work.

ModelOrientationViewpointCamera shot
Qwen3.5-2B17.2→32.417.2 \to32.433.3→45.233.3 \to45.240.9→49.740.9 \to49.7
Qwen3.5-9B20.7→34.320.7 \to34.341.3→46.941.3 \to46.942.4→48.842.4 \to48.8
Qwen3-VL-8B20.5→34.420.5 \to34.437.8→50.137.8 \to50.143.6→51.843.6 \to51.8
Gemma-4-E2B-it12.25→27.1712.25 \to27.1731.67→43.2531.67 \to43.2539.08→46.5839.08 \to46.58
Gemma-4-E4B-it16.83→28.4216.83 \to28.4240.00→46.3340.00 \to46.3341.08→49.5841.08 \to49.58
Gemma-4-26B-A4B-it16.83→34.0816.83 \to34.0843.42→47.3343.42 \to47.3342.33→51.4242.33 \to51.42
Gemma-4-31B-it14.08→34.0014.08 \to34.0040.67→43.9240.67 \to43.9243.33→49.4243.33 \to49.42
Avg. gain+15.2+7.8+7.8

Table J2. Accuracy (%) on the answer-balanced ImageNet3D evaluation set, reported as base →\to Ultimate3D-fine-tuned. Chance is 12.5% for orientation and 33.3% for viewpoint and camera shot.

Failure Analysis

We analyse the errors that remain after Ultimate3D fine-tuning on the synthetic Uti3D benchmark, the real-image PASCAL3D+ and MEBOW splits, and the ImageNet3D set of Appendix J. All ranges below are over the three fine-tuned models.

Which relations remain challenging. Table K3 gives per-class accuracy. On synthetic images accuracy is nearly uniform across classes, so the answer classes themselves are no longer a source of systematic error. On real photographs three relations stay hard: rear-oblique orientations (Back-Left/Back-Right, 42–61% on PASCAL3D+ against 74–85% for the pure side views; Back-Left→Left and Back-Right→Right alone account for 117 of Qwen3.5-9B’s 449 PASCAL3D+ orientation errors), the Bottom viewpoint (41–59% on PASCAL3D+, 25–47% on ImageNet3D, mostly answered Horizontal), and Long-shot framing on real photographs (14–22%), where our clean object-centred renders transfer poorly to cluttered wide shots. People (MEBOW) are the weakest object class overall, and only 2.3% of our synthetic training questions depict humans, none of them ordinary pedestrians.

Table K3: Per-class accuracy (%) after Ultimate3D fine-tuning; each cell is the range over Qwen3.5-2B, Qwen3.5-9B and Qwen3-VL-8B. PASCAL3D+ carries no camera-shot annotation and MEBOW only orientation. Rear-oblique orientations, Bottom and Long-shot are the residual weak classes.

RelationClassSyntheticPASCAL3D+MEBOWImageNet3D
OrientationBack79–9148–6248–6828–43
OrientationBack Left76–8943–6145–7026–43
OrientationLeft87–8974–8533–4735–53
OrientationFront Left77–9054–7329–3924–35
OrientationFront83–8966–7128–3429–39
OrientationFront Right72–8562–6930–3920–48
OrientationRight87–8875–8546–5840–41
OrientationBack Right71–9342–5728–4713–37
OrientationOverall82–8659–6437–3832–34
ViewpointTop70–8552–64–42–61
ViewpointHorizontal63–9166–76–47–59
ViewpointBottom57–7241–59–25–47
ViewpointOverall67–8358–62–45–50
Camera shotClose-up89–96––82–90
Camera shotMedium-shot83–88––40–52
Camera shotLong-shot88–91––14–22
Camera shotOverall87–92––49–52

Table K3.

Residual errors are one bin off. Fine-tuning changes the shape of the errors, not just their number: on PASCAL3D+ the share of orientation errors landing in a bin adjacent to the truth rises from 38–55% (base) to 63–64%, while gross (90∘90^\circ) errors fall from 15–30% to 8–15%. Tolerating a one-bin error raises PASCAL3D+ accuracy from 59–64% to 85–87% and synthetic accuracy from 82–86% to 91–93% (base models reach only 55–67% and 69–75% under the same tolerance). A second mode sits at 180∘180^\circ (20–22% of synthetic and 8–16% of PASCAL3D+ orientation errors) and concentrates on objects whose silhouette hides front from back — fans, monitors, sunglasses, and boats, buses or trains seen end-on (Figure K7, row 2).

Representative failures, part 1 of 2: geometric cues in the image plane

Figure K7. Representative failures, part 1 of 2: geometric cues in the image plane. Every image was answered incorrectly by all three fine-tuned models; panels give the ground-truth label (green) and the answer of Qwen3.5-9B + Ultimate3D (red). Boundary: the true azimuth is within a few degrees of a bin edge. Symmetry: front and back are barely distinguishable and the prediction is inverted. Rear-oblique: a shallow rear three-quarter view is answered with the neighbouring side bin. Elevation, framing and people failures are shown in Figure K8.

Representative failures showing viewpoint, framing, and people

Figure K8. Fig. K8: Representative failures, part 2 of 2: viewpoint, framing and people; continued from Figure K7. Every image was answered incorrectly by all three fine-tuned models; panels give the ground-truth label (green) and the answer of Qwen3.5-9B + Ultimate3D (red). Elevation: the camera is above or below but the cue is faint. Framing: the object fills the frame uncropped, at the Close-up/Medium-shot threshold. People: clutter, small scale, or a head facing away from the torso.

Errors concentrate at the label boundaries. Our benchmark discretises continuous camera parameters, and the residual errors are largely poses near a class boundary: using the exact render parameters, synthetic accuracy falls to 69–71% for orientations within 8∘8^\circ of a 45∘45^\circ bin edge, to 54–59% for elevations within 5∘5^\circ of the ±15∘\pm15^\circ band edge and to 36–44% for Medium-shots within 0.15 of the Close-up threshold, against 70–93% away from those boundaries; on ImageNet3D, whose azimuths are human-annotated, accuracy falls from 39–41% to 25–28% within 8∘8^\circ of a boundary, and from 35–37% to 15–20% for objects covering less than 30% of the frame.

Image conditions. We annotated 456 images: per (source, relation) group, 30 that all three finetuned models answered incorrectly, 20 answered incorrectly by one or two of them, and 20 answered correctly by all three as a matched control. Two annotators labelled every image independently and blind to all model predictions, with 80–99% agreement per field. Table K4 compares the shared failures with the matched successes. Failures are enriched for subtle geometric cues, near-boundary poses, clutter, small objects and shape-symmetric objects, but not for occlusion or truncation (33% vs. 32% on real photographs): partial visibility is not the limiting factor, cue strength is. Person images behave differently, with clutter the

Real photographsSynthetic
ConditionFSFS
Subtle directional / elevation cue52274128
Pose near a class boundary47284127
Cluttered background37181813
Small object (< 15% of frame)24153620
Shape-symmetric object9263
Unusual or articulated pose13121713
Multiple instances of the category121047
Occluded or truncated by the frame33322343
Ground-truth label judged questionable3803415

Table K4. Prevalence (%) of image conditions among images all three fine-tuned models answered incorrectly (F) and matched images all three answered correctly (S); n=90n = 90 and n=60n = 60 per column pair. Real = PASCAL3D+ and MEBOW.

Additional Qualitative Results with LLaVA

Figure H5. Additional Qualitative Results with LLaVA. We evaluate camera-object relation recognition for GPT-4o, Claude-3-Sonnet, LLaVA-1.6-13B, and finetuned LLaVA-1.6-13B on the Ultimate3D benchmark. Each model is asked questions, shown in gray boxes, about object orientation, camera viewpoint, and camera-shot type, together with the input images on the left. The responses show that finetuning LLaVA-1.6-13B on Ultimate3D substantially improves camera-object relation recognition compared with the base LLaVA model and closed-source models.

strongest signal (43% vs. 0%) alongside head–torso direction conflicts — conditions absent from our single-object renders.

Label ambiguity. The last row of Table K4 is a limitation of the benchmarks. On orientation the annotators reproduced the ground-truth label for 83–93% of the shared failures, so those are genuine model errors; on viewpoint and camera shot only 43–70%, because the discretisation admits no visual resolution (16∘16^\circ of elevation is Top, 14∘14^\circ is Horizontal). We therefore also report adjacent-bin accuracy for orientation and note the ±15∘\pm15^\circ elevation band and the fixed distance thresholds as design choices whose boundaries are inherently ambiguous.

Additional Discussions

We provide some insights of using synthetic generated visual instruction dataset for MLLM fine-tuning. For the task of camera-object relation recognition for 100 categories (as we collected in Ultimate3D dataset), a dataset comprising 100K to 1M VQAs will significantly improve baseline MLLMs by one-epoch finetuning. Further scaling up the dataset volume, finetuneing epochs, or model parameters may yield saturation.

Moreover, the quality of synthetic dataset is crucial to the improvement margin. Using dataset with higher success rate (ours is 93.07% as main paper § 4.7) provides better performances than using dataset generated other alternative backbones (i.e. SD-V1.5).

The coverage of arbitrary camera-object relation is also important for generalization. Our preliminary test using real images which are highly biased to front-facing direction from Pascal3D+ [86], results in poor generalization on uncovered orientations and viewpoints. This illustrates the importance of using synthetic dataset to eliminate the dataset biases.

For numerical prediction of camera-object relation, our preliminary test shows plausible success using MEBOW [85] dataset to improve MLLMs on predicting human orientation angles. The average degree prediction accuracy is competitive to the original paper [85]. However, the extended numerical prediction of 100 categories and 3 types of camera-object relations may require even larger dataset volume than Ultimate3D. Our framework is able to scale up the dataset volume. But due to the limitation of computing power, we leave it as interesting future work.

Algorithm of 3D visual instruction dataset generation pipeline

Algorithm 1 Synthetic VQA generation

Input: 3D asset A\mathcal{A}, asset category cc, camera-object relation β\beta.

Parameter: Renderer R\mathcal{R}, DM-based image generator G\mathcal{G}, image decoder D\mathcal{D}, LLM text generator L\mathcal{L}, system prompt given to L\mathcal{L} for generating image context description pimgp_{img}, and for generating QA pairs pqap_{qa}, DM sampling steps TT.

Output: Generated synthetic image IsynI_{syn}, corresponding text QA pairs Tqa\mathcal{T}_{qa}.

  1. Iβ=R(A,β)I_{\beta} = \mathcal{R}(\mathcal{A}, \beta) \hspace{1em} # Render 3D priors

  2. Timg=L(c,pimg)\mathcal{T}_{img} = \mathcal{L}(c, p_{img}) \hspace{1em} # Generate image description

  3. zT∼N(0,I)z_T \sim\mathcal{N}(0, I)

  4. for t=T,⋯ ,1t = T, \cdots, 1 do

  5. zt−1=G(zt,Iβ,Timg)z_{t-1} = \mathcal{G}(z_t, I_{\beta}, \mathcal{T}_{img}) \hspace{1em} # DM denoising

  6. end for

  7. Isyn=D(z0)I_{syn} = \mathcal{D}(z_0)

  8. Tqa=L(c,β,pqa)\mathcal{T}_{qa} = \mathcal{L}(c, \beta, p_{qa}) \hspace{1em} # Generate QA pairs

  9. return Isyn,TqaI_{syn}, \mathcal{T}_{qa}

References

  1. [1]Dai, W., Li, J., Li, D., Tiong, A., Zhao, J., Wang, W., Li, B., Fung, P.N., Hoi, S.: Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in neural information processing systems 36, 49250–49267 (2023)arxiv.org/abs/2305.06500
  2. [2]Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In: International Conference on Machine Learning, pp. 19730–19742 (2023). PMLRarxiv.org/abs/2301.12597
  3. [3]Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. Advances in neural information processing systems 36 (2024)arxiv.org/abs/2304.08485
  4. [4]Liu, H., Li, C., Li, Y., Lee, Y.J.: Improved baselines with visual instruction tuning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26296–26306 (2024)arxiv.org/abs/2310.03744
  5. [5]Li, F., Zhang, R., Zhang, H., Zhang, Y., Li, B., Li, W., Ma, Z., Li, C.: Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895 (2024)arxiv.org/abs/2407.07895
  6. [6]Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al.: Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185–24198 (2024)
  7. [7]Lin, J., Yin, H., Ping, W., Molchanov, P., Shoeybi, M., Han, S.: Vila: On pre-training for visual language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26689–26699 (2024)arxiv.org/abs/2312.07533
  8. [8]Tong, S., Brown, E., Wu, P., Woo, S., Middepogu, M., Akula, S.C., Yang, J., Yang, S., Iyer, A., Pan, X., et al.: Cambrian-1: A fully open, vision-centric exploration of multi-modal llms. arXiv preprint arXiv:2406.16860 (2024)arxiv.org/abs/2406.16860
  9. [9]Meta: Llama 3.2 Vision. https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/ (2024)
  10. [10]Google DeepMind: Gemma 4 Model Card. https://ai.google.dev/gemma/docs/core/model_card_4. Accessed: 2026-05-17 (2026)
  11. [11]Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., et al.: Qwen3-vl technical report. arXiv preprint arXiv:2511.21631 (2025)arxiv.org/abs/2511.21631
  12. [12]Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al.: Qwen3 technical report. arXiv preprint arXiv:2505.09388 (2025)arxiv.org/abs/2505.09388
  13. [13]Dosovitskiy, A.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)arxiv.org/abs/2010.11929
  14. [14]Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al.: Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023)arxiv.org/abs/2304.07193
  15. [15]Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J.E., Stoica, I., Xing, E.P.: Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90% ChatGPT Quality (2023). https://lmsys.org/blog/2023-03-30-vicuna/
  16. [16]Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)arxiv.org/abs/2302.13971
  17. [17]Google DeepMind: Gemini. https://deepmind.google/models/gemini/. Accessed: 2026-05-22 (2026) Singh, A., Fry, A., Perelman, A., Tart, J., Ganesh, A., El-Kishky, A., McLaughlin, A., Low, A., Ostrow, A., Anantharam, A., Nathan, A., Luo, A., Helyar, A., Madry, A., Efremov, A., Spyra, A., Baker-Whitcomb, A., Beutel, A., Karpenko, A., Makelov, A., Neitz, A., Wei, A., Barr, A., Kirchmeyer, A., Ivanov, A., Christakis, A., Gillespie, A., Tam, A., Bennett, A., Wan, A., Huang, A., Sandjideh, A.M., Yang, A., Kumar, A., Saraiva, A., Vallone, A., Gheorghe, A., Garcia, A.G., Braunstein, A., Liu, A., Schmidt, A., Mereskin, A., Mishchenko, A., Applebaum, A., Rogerson, A., Rajan, A., Wei, A., Kotha, A., Srivastava, A., Agrawal, A., Vijavvergiya, A., Tyra, A., Nair, A., Nayak, A., Eggers, B., Ji, B., Hoover, B., Chen, B., Chen, B., Barak, B., Minaiev, B., Hao, B., Baker, B., Lightcap, B., McKinzie, B., Wang, B., Quinn, B., Fioca, B., Hsu, B., Yang, B., Yu, B., Zhang, B., Brenner, B., Zetino, C.R., Raymond, C., Lugaresi, C., Paz, C., Hudson, C., Whitney, C., Li, C., Chen, C., Cole, C., Voss, C., Ding, C., Shen, C., Huang, C., Colby, C., Hallacy, C., Koch, C., Lu, C., Kaplan, C., Kim, C., Minott-Henriques, C., Frey, C., Yu, C., Czarnicki, C., Reid, C., Wei, C., Decareaux, C., Scheau, C., Zhang, C., Forbes, C., Tang, D., Goldberg, D., Roberts, D., Palmie, D., Kappler, D., Levine, D., Wright, D., Leo, D., Lin, D., Robinson, D., Grabb, D., Chen, D., Lim, D., Salama, D., Bhattacharjee, D., Tsipras, D., Li, D., Yu, D., Strouse, D., Williams, D., Hunn, D., Bayes, E., Arbus, E., Akyurek, E., Le, E.Y., Widmann, E., Yani, E., Proehl, E., Sert, E., Cheung, E., Schwartz, E., Han, E., Jiang, E., Mitchell, E., Sigler, E., Wallace, E., Ritter, E., Kavanaugh, E., Mays, E., Nikishin, E., Li, F., Such, F.P., Avila Belbute Peres, F., Raso, F., Bekerman, F., Tsimpourlas, F., Chatzis, F., Song, F., Zhang, F., Raila, G., McGrath, G., Briggs, G., Yang, G., Parascandolo, G., Chabot, G., Kim, G., Zhao, G., Valiant, G., Leclerc, G., Salman, H., Wang, H., Sheng, H., Jiang, H., Wang, H., Jin, H., Sikchi, H., Schmidt, H., Aspegren, H., Chen, H., Qiu, H., Lightman, H., Covert, I., Kivlichan, I., Silber, I., Sohl, I., Hammoud, I., Clavera, I., Lan, I., Akkaya, I., Kostrikov, I., Kofman, I., Ettinger, I., Singal, I., Hehir, J., Huh, J., Pan, J., Wilczynski, J., Pachocki, J., Lee, J., Quinn, J., Kiros, J., Kalra, J., Samaroo, J., Wang, J., Wolfe, J., Chen, J., Wang, J., Harb, J., Han, J., Wang, J., Zhao, J., Chen, J., Yang, J., Tworek, J., Chand, J., Landon, J., Liang, J., Lin, J., Liu, J., Wang, J., Tang, J., Yin, J., Jang, J., Morris, J., Flynn, J., Ferstad, J., Heidecke, J., Fishbein, J., Hallman, J., Grant, J., Chien, J., Gordon, J., Park, J., Liss, J., Kraaijeveld, J., Guay, J., Mo, J., Lawson, J., McGrath, J., Vendrow, J., Jiao, J., Lee, J., Steele, J., Wang, J., Mao, J., Chen, K., Hayashi, K., Xiao, K., Salahi, K., Wu, K., Sekhri, K., Sharma, K., Singhal, K., Li, K., Nguyen, K., Gu-Lemberg, K., King, K., Liu, K., Stone, K., Yu, K., Ying, K., Georgiev, K., Lim, K., Tirumala, K., Miller, K., Ahmad, L., Lv, L., Clare, L., Fauconnier, L., Itow, L., Yang, L., Romaniuk, L., Anise, L., Byron, L., Pathak, L., Maksin, L., Lo, L., Ho, L., Jing, L., Wu, L., Xiong, L., Mamit-suka, L., Yang, L., McCallum, L., Held, L., Bourgeois, L., Engstrom, L., Kuhn, L., Feurvrier, L., Zhang, L., Switzer, L., Kondraciuk, L., Kaiser, L., Joglekar, M., Singh, M., Shah, M., Stratta, M., Williams, M., Chen, M., Sun, M., Cayton, M., Li, M., Zhang, M., Aljubeh, M., Nichols, M., Haynes, M., Schwarzer, M., Gupta, M., Shah, M., Guan, M.Y., Huang, M., Dong, M., Wang, M., Glaese, M., Carroll, M., Lampe, M., Malek, M., Sharman, M., Zhang, M., Wang, M., Pokrass, M., Florian, M., Pavlov, M., Wang, M., Chen, M., Wang, M., Feng, M., Bavarian, M., Lin, M., Abdool, M., Rohaninejad, M., Soto, N., Staudacher, N., LaFontaine, N., Marwell, N., Liu, N., Preston, N., Turney, N., Ansman, N., Blades, N., Pancha, N., Mikhaylin, N., Felix, N., Handa, N., Rai, N., Keskar, N., Brown, N., Nachum, O., Boiko, O., Murk, O., Watkins, O., Gleeson, O., Mishkin, P., Lesiewicz, P., Baltescu, P., Belov, P., Zhokhov, P., Pronin, P., Guo, P., Thacker, P., Liu, Q., Yuan, Q., Liu, Q., Dias, R., Puckett, R., Arora, R., Mullapudi, R.T., Gaon, R., Miyara, R., Song, R., Agarwal, R., Marsan, R., Yemiuri, R., Xiong, R., Kshirsagar, R., Nuttall, R., Tsiupa, R., Eldan, R., Wang, R., James, R., Ziv, R., Shu, R., Nigmatullin, R., Jain, S., Talaie, S., Altman, S., Arnesen, S., Toizer, S., Toyer, S., Misenendino, S., Agarwal, S., Yoo, S., Heon, S., Ethersmith, S., Grove, S., Taylor, S., Bubeck, S., Banesiu, S., Amdo, S., Zhao, S., Wu, S., Santurkar, S., Zhao, S., Chaudhuri, S.R., Krishnaswamy, S., Shuaigi, Xia, Chen, S., Andkat, S., Fishman, S.P., Tobin, S., Fu, S., Jain, S., Mei, S., Egoian, S., Kim, S., Golden, S., Mah, S., Lin, S., Imm, S., Sharpe, S., Yadlowsky, S., Choudhry, S., Eum, S., Sanjeev, S., Khan, T., Stramer, T., Wang, T., Xin, T., Gogineni, T., Christianson, T., Sanders, T., Patwardhan, T., Degry, T., Shadwell, T., Fu, T., Gao, T., Garipov, T., Sriskandarajah, T., Sherbakov, T., Korbak, T., Kaftan, T., Hiratsuka, T., Wang, T., Song, T., Zhao, T., Peterson, T., Kharitonov, V., Chernova, V., Kosaraju, V., Kuo, V., Pong, V., Verma, V., Petrov, V., Jiang, W., Zhang, W., Zhou, W., Xie, W., Zhang, W., McCabe, W., DePue, W., Ellsworth, W., Bain, W., Thompson, W., Chen, X., Qi, X., Xiang, X., Shi, X., Dubois, Y., Yu, Y., Khakbaz, Y., Wu, Y., Qian, Y., Lee, Y.T., Chen, Y., Zhang, Y., Xiong, Y., Tian, Y., Chen, Y., Cha, Y., Bai, Y., Yang, Y., Yuan, Y., Li, Y., Zhang, Y., Yang, Y., Jin, Y., Jiang, Y., Wang, Y., Wang, Y., Liu, Y., Stubenvoll, Z., Dou, Z., Wu, Z., Wang, Z.: OpenAI GPT-5 System Card (2026). https://arxiv.org/abs/2601.03267
  18. [19]Anthropic: Introducing Claude Opus 4.7. https://www.anthropic.com/news/claude-opus-4-7. Accessed: 2026-05-22 (2026)
  19. [20]Yang, Z., Li, L., Lin, K., Wang, J., Lin, C.-C., Liu, Z., Wang, L.: The dawn of lmms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309.17421 9(1), 1 (2023)arxiv.org/abs/2309.17421
  20. [21]Tong, S., Liu, Z., Zhai, Y., Ma, Y., LeCun, Y., Xie, S.: Eyes wide shut? exploring the visual shortcomings of multimodal llms. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9568–9578 (2024)arxiv.org/abs/2401.06209
  21. [22]Fu, X., Hu, Y., Li, B., Feng, Y., Wang, H., Lin, X., Roth, D., Smith, N.A., Ma, W.-C., Krishna, R.: Blink: Multimodal large language models can see but not perceive. arXiv preprint arXiv:2404.12390 (2024)arxiv.org/abs/2404.12390
  22. [23]Yuksekgonul, M., Bianchi, F., Kalluri, P., Jurafsky, D., Zou, J.: When and why vision-language models behave like bags-of-words, and what to do about it? arXiv preprint arXiv:2210.01936 (2022)
  23. [24]Cheng, A.-C., Yin, H., Fu, Y., Guo, Q., Yang, R., Kautz, J., Wang, X., Liu, S.: Spatialrgpt: Grounded spatial reasoning in vision language model. arXiv preprint arXiv:2406.01584 (2024)arxiv.org/abs/2406.01584
  24. [25]Chen, B., Xu, Z., Kirmani, S., Ichter, B., Sadigh, D., Guibas, L., Xia, F.: Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14455–14465 (2024)arxiv.org/abs/2401.12168
  25. [26]Guo, Q., De Mello, S., Yin, H., Byeon, W., Cheung, K.C., Yu, Y., Luo, P., Liu, S.: Regiongpt: Towards region understanding vision language model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13796–13806 (2024)arxiv.org/abs/2403.02330
  26. [27]Chen, G.H., Chen, S., Zhang, R., Chen, J., Wu, X., Zhang, Z., Chen, Z., Li, J., Wan, X., Wang, B.: Allava: Harnessing gpt4v-synthesized data for a lite vision-language model. arXiv preprint arXiv:2402.11684 (2024)arxiv.org/abs/2402.11684
  27. [28]Li, X., Tu, H., Hui, M., Wang, Z., Zhao, B., Xiao, J., Ren, S., Mei, J., Liu, Q., Zheng, H., et al.: What if we recaption billions of web images with llama-3? arXiv preprint arXiv:2406.08478 (2024)arxiv.org/abs/2406.08478
  28. [29]Zhang, Y., Wu, J., Li, W., Li, B., Ma, Z., Liu, Z., Li, C.: Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713 (2024)arxiv.org/abs/2410.02713
  29. [30]Sharifzadeh, S., Kaplanis, C., Pathak, S., Kumaran, D., Ilic, A., Mitrovic, J., Blundell, C., Banino, A.: Synth2: Boosting visual-language models with synthetic captions and image embeddings. arXiv preprint arXiv:2403.07750 (2024)arxiv.org/abs/2403.07750
  30. [31]Taylor, S., Bubeck, S., Banesiu, S., Amdo, S., Zhao, S., Wu, S., Santurkar, S., Zhao, S., Chaudhuri, S.R., Krishnaswamy, S., Shuaigui, Xia, Chen, S., Andkdat, S., Fishman, S.P., Tobin, S., Fu, S., Jain, S., Mei, S., Egoian, S., Kim, S., Golden, S., Mah, S., Lin, S., Imm, S., Sharpe, S., Yadlowsky, S., Choudhry, S., Eum, S., Sanjeev, S., Khan, T., Stramer, T., Wang, T., Xin, T., Gogineni, T., Christianson, T., Sanders, T., Patwardhan, T., Degry, T., Shadwell, T., Fu, T., Gao, T., Garipov, T., Sriskandarajah, T., Sherbakov, T., Korbak, T., Kaftan, T., Hiratsuka, T., Wang, T., Song, T., Zhao, T., Peterson, T., Kharitonov, V., Chernova, V., Kosaraju, V., Kuo, V., Pong, V., Verma, V., Petrov, V., Jiang, W., Zhang, W., Zhou, W., Xie, W., Zhan, W., McCabe, W., DePue, W., Ellsworth, W., Bain, W., Thompson, W., Chen, X., Qi, X., Xiang, X., Shi, X., Dubois, Y., Yu, Y., Khakbaz, Y., Wu, Y., Qian, Y., Lee, Y.T., Chen, Y., Zhang, Y., Xiong, Y., Tian, Y., Cha, Y., Bai, Y., Yang, Y., Yuan, Y., Li, Y., Zhang, Y., Yang, Y., Jin, Y., Jiang, Y., Wang, Y., Wang, Y., Liu, Y., Stubenvoll, Z., Dou, Z., Wu, Z., Wang, Z.: OpenAI GPT-5 System Card (2026). https://arxiv.org/abs/2601.03267 Zhang, Y., Zhang, R., Gu, J., Zhou, Y., Lipka, N., Yang, D., Sun, T.: Llava\ r: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint arXiv:2306.17107 (2023)
  31. [32]Du, Y., Guo, H., Zhou, K., Zhao, W.X., Wang, J., Wang, C., Cai, M., Song, R., Wen, J.-R.: What makes for good visual instructions? synthesizing complex visual reasoning instructions for visual instruction tuning. arXiv preprint arXiv:2311.01487 (2023)arxiv.org/abs/2311.01487
  32. [33]Liu, Z., Liang, H., Xiong, W., Yu, Q., He, C., Cui, B., Zhang, W.: Synthvlm: High-efficiency and high-quality synthetic data for vision language models. arXiv preprint arXiv:2407.20756 (2024)
  33. [34]Du, Z., Li, H., Guo, X., Li, B.: Training on synthetic data beats real data in multimodal relation extraction. arXiv preprint arXiv:2312.03025 (2023)arxiv.org/abs/2312.03025
  34. [35]Li, Y., Zhang, C., Yu, G., Wang, Z., Fu, B., Lin, G., Shen, C., Chen, L., Wei, Y.: Stablellava: Enhanced visual instruction tuning with synthesized image-dialogue data. arXiv preprint arXiv:2308.10253 (2023)arxiv.org/abs/2308.10253
  35. [36]Johnson, J., Hariharan, B., Van Der Maaten, L., Fei-Fei, L., Lawrence Zitnick, C., Girshick, R.: Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2901–2910 (2017)arxiv.org/abs/1612.06890
  36. [37]Liu, R., Liu, C., Bai, Y., Yuille, A.L.: Clevr-ref+: Diagnosing visual reasoning with referring expressions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4185–4194 (2019)arxiv.org/abs/1901.00850
  37. [38]Greff, K., Belletti, F., Beyer, L., Doersch, C., Du, Y., Duckworth, D., Fleet, D.J., Gnanapragasam, D., Golemo, F., Herrmann, C., et al.: Kubric: A scalable dataset generator. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3749–3761 (2022)arxiv.org/abs/2203.03570
  38. [39]Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740–755 (2014). Springerarxiv.org/abs/1405.0312
  39. [40]Young, P., Lai, A., Hodosh, M., Hockenmaier, J.: From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics 2, 67–78 (2014)DOI
  40. [41]Yu, L., Poirson, P., Yang, S., Berg, A.C., Berg, T.L.: Modeling context in referring expressions. In: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pp. 69–85 (2016). Springerarxiv.org/abs/1608.00272
  41. [42]Hudson, D.A., Manning, C.D.: Gqa: A new dataset for real-world visual reasoning and compositional question answering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6700–6709 (2019)arxiv.org/abs/1902.09506
  42. [43]Liu, F., Emerson, G., Collier, N.: Visual spatial reasoning. Transactions of the Association for Computational Linguistics 11, 635–651 (2023)arxiv.org/abs/2205.00363
  43. [44]Gurari, D., Li, Q., Stangl, A.J., Guo, A., Lin, C., Grauman, K., Luo, J., Bigham, J.P.: Vizwiz grand challenge: Answering visual questions from blind people. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3608–3617 (2018)arxiv.org/abs/1802.08218
  44. [45]Wang, J., Meng, L., Weng, Z., He, B., Wu, Z., Jiang, Y.-G.: To see is to believe: Prompting gpt-4v for better visual instruction tuning. arXiv preprint arXiv:2311.07574 (2023)arxiv.org/abs/2311.07574
  45. [46]Marino, K., Rastegari, M., Farhadi, A., Mottaghi, R.: Ok-vqa: A visual question answering benchmark requiring external knowledge. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3195–3204 (2019)arxiv.org/abs/1906.00067
  46. [47]Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., Kalyan, A.: Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems 35, 2507–2521 (2022)arxiv.org/abs/2209.09513
  47. [48]Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., Parikh, D.: Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6904–6913 (2017)arxiv.org/abs/1612.00837
  48. [49]Mishra, A., Shekhar, S., Singh, A.K., Chakraborty, A.: Ocr-vqa: Visual question answering by reading text in images. In: 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 947–952 (2019). IEEEDOI
  49. [50]Singh, A., Natarjan, V., Shah, M., Jiang, Y., Chen, X., Parikh, D., Rohrbach, M.: Towards vqa models that can read. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8317–8326 (2019)arxiv.org/abs/1904.08920
  50. [51]Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al.: Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems 35, 25278–25294 (2022)
  51. [52]Gupta, A., Dollar, P., Girshick, R.: Lvis: A dataset for large vocabulary instance segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5356–5364 (2019)arxiv.org/abs/1908.03195
  52. [53]Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Krawitz, J., Chen, S., Kalanditis, Y., Li, L.-J., Shamma, D.A., et al.: Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision 123, 32–73 (2017)arxiv.org/abs/1602.07332
  53. [54]Zhou, B., Zhao, H., Puig, X., Xiao, T., Fidler, S., Barriuso, A., Torralba, A.: Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision 127(3), 302–321 (2019)arxiv.org/abs/1608.05442
  54. [55]OpenAI: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)arxiv.org/abs/2303.08774
  55. [56]Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in neural information processing systems 33, 6840–6851 (2020)arxiv.org/abs/2006.11239
  56. [57]Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 (2020)arxiv.org/abs/2010.02502
  57. [58]Song, Y., He, L., Zhang, Z., Kim, S.Y., Zhang, H., Xiong, W., Lin, Z., Price, B.L., Cohen, S., Zhang, J., et al.: Refine-by-align: Reference-guided artifacts refinement through semantic alignment. In: The Thirteenth International Conference on Learning Representations (2025)arxiv.org/abs/2412.00306
  58. [59]Song, Y., Zhang, Z., Lin, Z., Cohen, S., Price, B., Zhang, J., Kim, S.Y., Zhang, H., Xiong, W., Aliaga, D.: Imprint: Generative object compositing by learning identity-preserving representation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8048–8058 (2024)arxiv.org/abs/2403.10701
  59. [60]He, L., Lu, Y., Corring, J., Florencio, D., Zhang, C.: Diffusion-based document layout generation. In: International Conference on Document Analysis and Recognition, pp. 361–378 (2023). SpringerDOI
  60. [61]He, L., Song, Y., Huang, H., Liu, P., Tang, Y., Aliaga, D., Zhou, X.: Kubrick: Multimodal agent collaborations for synthetic video generation. arXiv preprint arXiv:2408.10453 (2024)arxiv.org/abs/2408.10453
  61. [62]Song, Y., Zhang, Z., Lin, Z., Cohen, S., Price, B., Zhang, J., Kim, S.Y., Aliaga, D.: Object-stitch: Object compositing with diffusion model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18310–18319 (2023)
  62. [63]Liao, Y.-H., Elflein, S., He, L., Leal-Taixé, L., Choi, Y., Fidler, S., Acuna, D.: Longperceptualthoughts: Distilling system-2 reasoning for system-1 perception. arXiv preprint arXiv:2504.15362 (2025)arxiv.org/abs/2504.15362
  63. [64]Hua, H., Zeng, Z., Song, Y., Tang, Y., He, L., Aliaga, D., Xiong, W., Luo, J.: Mmig-bench: Towards comprehensive and explainable evaluation of multi-modal image generation models. arXiv preprint arXiv:2505.19415 (2025)arxiv.org/abs/2505.19415
  64. [65]Sun, L., He, L., Jia, S., He, Y., You, C.: Docagent: An agentic framework for multi-modal long-context document understanding. In: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 17712–17727 (2025)
  65. [66]Tang, Y., Bi, J., Huang, C., Liang, S., Shimada, D., Hua, H., Xiao, Y., Song, Y., Liu, P., Feng, M., et al.: Caption anything in video: Fine-grained object-centric captioning via spatiotemporal multimodal prompting. arXiv preprint arXiv:2504.05541 (2025)arxiv.org/abs/2504.05541
  66. [67]He, L., Aliaga, D.: Coho: Context-sensitive city-scale hierarchical urban layout generation. In: European Conference on Computer Vision, pp. 1–18 (2024). Springer
  67. [68]Tang, Y., Guo, J., Liu, P., Wang, Z., Hua, H., Zhong, J.-X., Xiao, Y., Huang, C., Song, L., Liang, S., et al.: Generative ai for cel-animation: A survey. arXiv preprint arXiv:2501.06250 (2025)arxiv.org/abs/2501.06250
  68. [69]He, L., Aliaga, D.: Globalmapper: Arbitrary-shaped urban layout generation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 454–464 (2023)arxiv.org/abs/2307.09693
  69. [70]He, L., Shan, J., Aliaga, D.: Generative building feature estimation from satellite images. IEEE Transactions on Geoscience and Remote Sensing 61, 1–13 (2023)
  70. [71]Yuan, Y., Wang, X., Wickremasinghe, T., Nadir, Z., Ma, B., Chan, S.H.: Newton-gen: Physics-consistent and controllable text-to-video generation via neural newtonian dynamics. arXiv preprint arXiv:2509.21309 (2025)
  71. [72]Yuan, Y., Wang, X., Sheng, Y., Chennuri, P., Zhang, X., Chan, S.: Generative photography: Scene-consistent camera control for realistic text-to-image synthesis. In: Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 7920–7930 (2025)
  72. [73]Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684–10695 (2022)
  73. [74]Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., Rombach, R.: Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 (2023)
  74. [75]Krojer, B., Vattikonda, D., Lara, L., Jampani, V., Portelance, E., Pal, C., Reddy, S.: Learning action and reasoning-centric image editing from videos and simulations. arXiv preprint arXiv:2407.03471 (2024)
  75. [76]Wang, S., Saharia, C., Montgomery, C., Pont-Tuset, J., Noy, N.S., Pellegrini, S., Onofri, Y., Laszlo, S., Fleet, D.J., Soricut, R., et al.: Imagen editor and editbench: Advancing and evaluating text-guided image inpainting. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18359–18369 (2023)
  76. [77]Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., Cohen-Or, D.: Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626 (2022)
  77. [78]Zhang, K., Mo, L., Chen, W., Sun, H., Su, Y.: Magicbrush: A manually annotated dataset for instruction-guided image editing. Advances in Neural Information Processing Systems 36 (2024)
  78. [79]Zhang, L., Rao, A., Agrawala, M.: Adding conditional control to text-to-image diffusion models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836–3847 (2023)arxiv.org/abs/2302.05543
  79. [80]Ma, W., Liu, Q., Wang, J., Wang, A., Yuan, X., Zhang, Y., Xiao, Z., Zhang, G., Lu, B., Duan, R., et al.: Generating images with 3d annotations using diffusion models. In: The Twelfth International Conference on Learning Representations (2024)arxiv.org/abs/2306.08103
  80. [81]Deitke, M., Schwenk, D., Salvador, J., Weihs, L., Michel, O., Vanderbilt, E., Schmidt, L., Ehsani, K., Kembhavi, A., Farhadi, A.: Objaverse: A universe of annotated 3d objects. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13142–13153 (2023)arxiv.org/abs/2212.08051
  81. [82]Deitke, M., Liu, R., Wallingford, M., Ngo, H., Michel, O., Kusupati, A., Fan, A., Laforte, C., Voleti, V., Gadre, S.Y., et al.: Objaverse-XL: A universe of 10m+ 3d objects. Advances in Neural Information Processing Systems 36 (2024)arxiv.org/abs/2307.05663
  82. [83]Chang, A.X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., et al.: Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012 (2015)arxiv.org/abs/1512.03012
  83. [84]Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.: Imagenet large scale visual recognition challenge. International journal of computer vision 115, 211–252 (2015)arxiv.org/abs/1409.0575
  84. [85]Wu, C., Chen, Y., Luo, J., Su, C.-C., Dawane, A., Hanzra, B., Deng, Z., Liu, B., Wang, J.Z., Kuo, C.-h.: Mebow: Monocular estimation of body orientation in the wild. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3451–3461 (2020)arxiv.org/abs/2011.13688
  85. [86]Xiang, Y., Mottaghi, R., Savarese, S.: Beyond pascal: A benchmark for 3d object detection in the wild. In: IEEE Winter Conference on Applications of Computer Vision (WACV) (2014)
  86. [87]Xu, J., Liu, X., Wu, Y., Tong, Y., Li, Q., Ding, M., Tang, J., Dong, Y.: ImageReward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems 36 (2024)arxiv.org/abs/2304.05977
  87. [88]Hessel, J., Holtzman, A., Forbes, M., Bras, R.L., Choi, Y.: Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718 (2021)arxiv.org/abs/2104.08718
  88. [89]Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems 30 (2017)arxiv.org/abs/1706.08500
  89. [90]Ma, W., Zeng, G., Zhang, G., Liu, Q., Zhang, L., Kortylewski, A., Liu, Y., Yuille, A.: Imagenet3d: Towards general-purpose object-level 3d understanding. arXiv preprint arXiv:2406.09613 (2024)arxiv.org/abs/2406.09613

Paper details

vlnc:W_8cx3fsz5
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents