betaFeedback

WorksComputer Science

One Code per Material: Neural Representation and Generation of Measured BTFs

Abstract

Measured bidirectional texture functions (BTFs) capture the full appearance of real materials–including parallax, self-shadowing and inter-reflections that analytic SVBRDFs cannot express–but each measurement is billions of values, and every new material requires a new capture. Existing neural BTF representations encode each material as large per-texel latent maps rather than as a point in a shared space from which new materials could be sampled. We present neural representations of measured BTFs at two operating points. A per-material coordinate network outperforms the recent neural BTF of Xia et al. at matched size (+3.2 dB PSNR) and all prior baselines on every metric. A compact model represents each material by a single 256-dimensional code that conditions a shared decoder; it exceeds the PSNR (+1.2 dB) and colour accuracy of the per-texel latent maps of Rainer et al. with 256 numbers per material instead of millions, and one decoder shared by all materials. Because each material is a single point in this code space, measured BTFs can be sampled: a class-conditional flow over the codes, trained on the 84 materials enlarged with colourways and hue rotations, generates materials that render and relight as full BTFs.

From measured to sampled BTFs in one scene

Figure 1. From measured to sampled BTFs in one scene, rendered with Mitsuba 3 [17]. Every textured surface is a BTF reproduced or generated by our models: the per-material networks and the compact model represent measured materials, and the flow model samples further ones; only the paintings and plants use standard textures. Close-ups: towards grazing angles, the relief of leather, fabric and carpet keeps the measured shading and shadowing between neighbouring texels, an effect a per-texel SVBRDF cannot represent [5].

Introduction

Real materials owe much of their appearance to structure below the scale of a surface: the loops of a carpet, the crack network of leather, the weave of a fabric. Under changing light and view these structures cast shadows, occlude each other and shift with parallax. Bidirectional texture functions (BTFs) [5] capture this appearance directly by measuring a planar sample under many light and view directions, and measured BTF datasets such as UBO2014 [46] remain the reference for realistic material appearance. Analytic SVBRDFs, such as the metallic–roughness model of most material generators [3, 18], cannot represent such complex effects, which give real materials their realism (§ 4.1).

The price of this fidelity is size. A single UBO2014 material is 400×400400 \times400 texels under 151×151151 \times151 light–view pairs, about 101010^{10} values, and creating a new material requires a new capture. Neural BTF representations reduce the storage by decoding per-texel latent features with a small network: latent maps [33, 34], feature textures with neural offsets [22, 47], or feature planes [10]. These representations reconstruct well, but store a feature vector per texel and are not designed to sample new materials. Generative material models, on the other hand, learn from large synthetic corpora or from single flash photographs and produce parametric SVBRDFs [13, 28, 61] or neural materials fitted to synthetic data [32]; none of them learns from measured BTFs, of which only about a hundred are publicly available.

We bring the two together: a representation of measured BTFs that is accurate and compact, and whose codes can be sampled. We achieve this in three steps (Figure 2). A per-material coordinate network gives the most accurate representation. A compact model replaces the per-material network by a single global code per material and a shared decoder, so a material is described by 256 floats. Because each material is a point in this low-dimensional space, the codes also lend themselves to generation, which we explore with a flow model [25, 26] trained on the codes: it samples materials, and the same decoder renders them as full BTFs with measured angular behaviour.

Overview of the compact model, showing measured BTFs and colourways, a set encoder, sampled codes, a shared decoder, reconstructed samples, and relit samples rendered on objects

Figure 2. Overview. The compact model encodes each measured material and its colourways (§ 3.3) into one 256-float code with a set encoder; a decoder gθg_\theta shared by all materials maps the code, texel position and light and view directions to RGB. A class-conditional rectified flow fitted to these codes samples codes, which the same decoder renders as full BTFs that can be relit. Top right: measured materials reconstructed by the compact model, rendered on objects.

Measured BTF datasets are inherently small, with UBO2014 containing only twelve materials per class, making it difficult to learn a distribution of material appearances. To address this limitation, we augment the training set through colourway transfer, combining the measured structure of one material with the colour characteristics of another material from the same class. This expands the dataset from 84 to 886 materials, so that the generator learns colour variation over the measured structures.

In summary, we contribute:

  • a per-material neural BTF that outperforms all neural BTF baselines at comparable size (2.6–3.4M values per material);

  • a compact model with one 256-float code per material and a shared decoder that exceeds the PSNR and colour accuracy of the per-texel latent maps of Rainer et al. [34] with 14× fewer values per material, even counting the shared decoder;

  • a code space in which measured BTFs can be sampled: a class-conditional flow over the codes learns the distribution of 84 captures enlarged with colourways and hue rotations (released as an augmented dataset of 886 materials), and its samples render and relight as full BTFs.

Related Work

Measured BTFs and neural materials. BTFs were introduced by Dana et al. [5]; UBO2014 [46] provides 84 measured materials in seven classes at 151×151151 \times151 light–view pairs and is distributed as a rank-limited factorisation. Classic compression factorises the six-dimensional function linearly or sparsely [19]. Neural approaches replaced the factorisation with learned decoders: Rainer et al. [33] train a per-material autoencoder over per-texel angular profiles, and later a unified encoder and decoder shared across materials [34], which still stores a 32-dimensional latent per texel. NeuMIP [22] combines neural texture pyramids with learned neural offsets for parallax, and follow-up work refines its architecture [50], synthesises BTFs of arbitrary extent from an example [48, 49], and runs neural materials in real-time path tracers [55]; Dou et al. [7] share neural primitives across materials through a codebook. Neural Bi-planes [10] use a universal decoder with per-material feature planes; NeuBTF [35] conditions a neural BTF on a guidance image for tiling and transfer; and Xia et al. [47] fit multi-resolution neural textures with a neural offset per material and extend a single material to larger extents by tiling its latent texture. Concurrently, RoboCloth [24] captures 500 real cloth materials and trains a shared decoder with a dense latent texture per material. Idema and Peers [16] infer per-pixel neural material parameters from a single flash photograph. All of these store features per texel or per pixel. We compare against the official implementations of [10, 34, 47].

Latent codes, hypernetworks and generation in function space. Sztrajman et al. [39] represent a measured BRDF with a small MLP, an autoencoder gives a latent space for editing measured BRDFs [15], and hypernetworks map a latent code to such networks and give a space of BRDFs [11, 23], and physically based neural BRDFs enforce reciprocity and energy conservation for reconstruction and editing [59]. Fourier features [40] allow coordinate networks to fit high-frequency signals, and auto-decoders [30] learn one code per shape for a shared network. Generative models can then be trained on such neural fields: functa [8] fit a generative model to per-datum modulations of a shared network, HyperDiffusion [9] diffuses directly in MLP weight space, and M3^{3}ashy [58] applies hyperdiffusion to neural fields of measured, spatially uniform BRDFs. We extend the latent-code idea from spatially uniform BRDFs to spatially varying, measured BTFs, where the decoder must also represent the spatial layout, and train the generative model on one global code per material.

Generative material models. Most material generators produce SVBRDF maps for an analytic BRDF [20]. MaterialGAN [12] and TileGen [60] train StyleGAN generators on synthetic SVBRDFs; MatFuse [44], ControlMat [43], Material Palette [27] and StableMaterials [41] use diffusion models trained on synthetic material libraries such as MatSynth [42]. Generated synthetic materials often look synthetic, and several works learn from real photographs to close this gap: Henzler et al. [13] learn a generative space of SVBRDF textures from flash photographs, PhotoMat [61] trains a generator of neural material features on 12,000 real flash photos, and RealMat [62] finetunes a diffusion model with a realism reward learned from real material photographs. Beyond analytic BRDFs, a conditional GAN predicts BTF images of known materials under new light and view directions [57], MesoGAN [6] generates neural reflectance shells for mesoscale appearance, Generative Neural Materials [32] train a diffusion model over neural material textures from a large corpus of synthetic materials, and video-diffusion approaches generate or extract PBR and neural materials [28, 51, 52]. All of these learn from synthetic materials, from single photographs, or from the priors of image and video models. We learn from measured BTFs, which record the full light- and view-dependent appearance of real samples, including parallax, masking and self-shadowing.

Scarce data, augmentation and memorisation. Measured BTFs are scarce: UBO2014 has 84 materials, and a concurrent capture has 500 cloth samples [24]. Generative models reproduce their training data when it is small; even diffusion models trained on large image sets copy training images [4, 37]. Augmentation is therefore common: M3^{3}ashy [58] enlarges MERL by permuting colour channels and interpolating in PCA space, and Yu et al. [53] procedurally enrich simple PBR materials with layered, multilobe effects before training a generator. We instead recolour measured BTFs within their class with a linear colour transfer [31], so that every training value remains a measured value up to a 3×33 \times3 colour transform, and the colourways keep the structure, relief and shadowing of the measured BTFs.

Method

A BTF B(x,ωi;ωo)B(\mathbf{x},\boldsymbol{\omega}_i;\boldsymbol{\omega}_o) gives the reflected RGB radiance factor at texel x\mathbf{x} of a planar sample for light direction ωi\boldsymbol{\omega}_i and view direction ωo\boldsymbol{\omega}_o. Unlike an SVBRDF, it is measured rather than modelled, and therefore includes the sample’s meso-scale effects as they appear at each texel. We parameterise the texel position by x∈[−1,1]2\mathbf{x} \in[-1,1]^2, and the direction pair by the Rusinkiewicz half and difference vectors (h,d)∈R6(\mathbf{h},\mathbf{d}) \in\mathbb{R}^6 [36], keeping the azimuth because BTF texels are anisotropic. Figure 2 summarises our approach.

Per-material BTF

We first build a per-material representation: a Fourier-feature MLP fitted to a single material, which gives our most accurate results and later becomes the backbone of our compact model (§ 3.2). For each material mm, we fit

B^m(x,ωi,ωo)=exp⁡(gm([γ(x),h,d]))−1,(1)\hat{B}_m(\mathbf{x},\boldsymbol{\omega}_i,\boldsymbol{\omega}_o)=\exp(g_m([\gamma(\mathbf{x}),\mathbf{h},\mathbf{d}]))-1, \tag*{(1)}

where γ(x)=[sin⁡(2πFx),cos⁡(2πFx)]\gamma(\mathbf{x})=[\sin(2\pi\mathbf{F}\mathbf{x}),\cos(2\pi\mathbf{F}\mathbf{x})] are random Fourier features [40] with F∈R128×2\mathbf{F} \in\mathbb{R}^{128 \times2} drawn from N(0,162)\mathcal{N}(0,16^2), and gmg_m is an MLP with four hidden layers of width 1024 and ReLU activations (3.42M parameters). The exponential output follows neural BRDF representations [39] and keeps predictions in the dynamic range of measured reflectance.

We minimise a log-space L1L_1 loss on cosine-weighted values over nn random queries (x,ωi,ωo)(\mathbf{x},\boldsymbol{\omega}_i,\boldsymbol{\omega}_o) drawn from all texels and all measured direction pairs,

L=1n∑∣log⁡(1+cos⁡θiB^)−log⁡(1+cos⁡θiB)∣.(2)\mathcal{L}=\frac{1}{n}\sum\left|\log(1+\cos\theta_i\hat{B})-\log(1+\cos\theta_iB)\right|. \tag*{(2)}

The logarithm balances specular highlights against dark, shadowed texels.

Compact model

The compact model replaces the per-material network by one shared decoder gθg_{\theta} and a single code zm∈R256\mathbf{z}_m \in\mathbb{R}^{256} per material,

B^m(x,ωi,ωo)=exp⁡(gθ([γ(x),h,d,zm]))−1,(3)\hat{B}_m(\mathbf{x},\boldsymbol{\omega}_i,\boldsymbol{\omega}_o)=\exp(g_{\theta}([\gamma(\mathbf{x}),\mathbf{h},\mathbf{d},\mathbf{z}_m]))-1, \tag*{(3)}

with 512 Fourier frequencies drawn from N(0,322)\mathcal{N}(0,32^2) and four hidden layers of width 3072 (32.3M parameters, shared by all materials). We condition the decoder by concatenating the material code with each query.

Encoding. A permutation-invariant set encoder [54] infers zm\mathbf{z}_m from a set of kk measured samples {(xj,hj,dj,bj)}j=1k\{(\mathbf{x}_j,\mathbf{h}_j,\mathbf{d}_j,\mathbf{b}_j)\}_{j=1}^{k} with measured RGB values bj\mathbf{b}_j, applying an MLP fϕf_{\phi} to each sample and averaging,

zm=1k∑j=1kfϕ(xj,hj,dj,bj).(4)\mathbf{z}_m=\frac{1}{k}\sum_{j=1}^{k}f_{\phi}(\mathbf{x}_j,\mathbf{h}_j,\mathbf{d}_j,\mathbf{b}_j). \tag*{(4)}

The encoder maps a set of measured samples, each consisting of a spatial position, a light–view pair, and its corresponding colour, to a compact latent code. The decoder then uses this code, together with a spatial position and a light–view pair, to predict the corresponding colour. During training we draw a fresh random context of k=16,384k=16{,}384 samples per material and step; at test time we encode from all samples. Rendering only requires zmz_m and gθg_\theta.

Training. Each step samples four materials and, for each, n=65,536n=65{,}536 random queries, and minimises Eq. (2) plus λ∥zm∥22\lambda\lVert z_m\rVert_2^2 with λ=10−2\lambda=10^{-2}; the penalty keeps the codes small and on a common scale, so that the code space stays compact for the generative model. A single UBO2014 material contains 4002×15122×3≈1.1×1010400^2 \times1512^2 \times3 \approx1.1 \times10^{10} values; we train for 705k steps with cosine learning-rate decay, so each measured sample is seen only 0.6 times on average.

Colourways

UBO2014 has only twelve materials per class, which makes it difficult to learn a distribution of materials. We therefore enlarge the 84 measured materials with colourways (Figure 3): for two materials AA and BB of the same class, we recolour AA with the colours of BB. Since a BTF stores RGB reflectance rather than colour, one 3×33 \times3 matrix MABM_{AB} is applied to every measured RGB reflectance value of AA, at every texel and every light–view pair, so the result keeps the structure, relief and shadowing of AA while its colours change. Negative values, which the matrix can produce, are set to zero; values above one are kept, as the reflectance of a measured BTF can go above one. This mirrors how fabrics, leathers and wallpapers are produced in colourways: the same weave, grain or print in different dyes. We fit MABM_{AB} by linear colour transfer [31], mapping the mean and covariance of the colours in AA’s frontal image onto those of BB’s. In total, the 84 measured materials yield 1,141 training materials,

Measured materials and their colourways across carpet, fabric, felt, and wallpaper

Figure 3. Colourways. One measured material per row (left) and four of its colourways: its structure in the colours of other materials of its class, as flash photographs. Every value is a measured value under one 3×33 \times3 colour transform.

including hue rotations for felt and carpet (details in supplementary Sec. S3).

For generation, the compact model is finetuned on the augmented training set, starting from an earlier checkpoint (supplementary Sec. S1).

Generation

We learn the distribution of codes with a class-conditional rectified flow [25, 26]. Given normalised codes z1z_1 of training materials, their class labels c∈{1,…,7}c \in\{1,\ldots,7\}, the index of their UBO2014 class (carpet, fabric, felt, leather, stone, wallpaper or wood), and Gaussian noise z0z_0, the velocity network vϕv_\phi is trained with

Lflow=Et,z0,z1∥vϕ((1−t)z0+tz1,t,c)−(z1−z0)∥2(5)\mathcal{L}_{\mathrm{flow}}=\mathbb{E}_{t,z_0,z_1}\left\|v_\phi\left((1-t)z_0+tz_1,t,c\right)-(z_1-z_0)\right\|^2 \tag*{(5)}

dropping the class with probability 0.1 for classifier-free guidance [14]. vϕv_\phi is an MLP of width 512 with four residual blocks (4.1M parameters) whose scale, shift and gate are modulated by the time embedding and a learned embedding of the class label (adaptive layer normalisation). The flow is trained on the codes of 886 materials, leaving out the 255 colourways and hue rotations whose texture is faint (supplementary Sec. S3). At test time we integrate z˙=vu+w(vc−vu)\dot{z}=v_u+w(v_c-v_u) from noise, where vcv_c and vuv_u are the velocities with and without the class, with 100 Euler steps and guidance weight w=2w=2, and decode the resulting code with gθg_\theta into a full BTF that can be evaluated at any texel, light and view.

Experiments

Data. We use all 84 materials of UBO2014 [46], seven classes of materials found in indoor scenes (carpet, fabric, felt, leather, stone, wallpaper, wood) with twelve material instances each. Every material is a 5 cm ×\times 5 cm planar sample captured fully automatically under directional illumination as 22,801 HDR images, one for every combination of 151 view and 151 light directions. The images are resampled to 400 ×\times 400 texels and distributed as a rank-101 factorisation per material; we decode this factorisation and use it as the reference for training and evaluating all methods.

Protocol. All methods are evaluated on nine materials covering every class (carpet01, fabric01/05/07, felt01, leather01, stone01, wallpaper01, wood01) with a single shared protocol, whose results are reported in Tab. 1: each method predicts the BTF directly at eight light–view pairs spread evenly over the measured grid, from frontal to the grazing limit of the capture (light and view at 75° elevation), and is compared with the measured images at the same pairs; one exposure per material, shared by all methods; PSNR, SSIM and LPIPS [56] on tonemapped images; and ΔEITP\Delta E_{\mathrm{ITP}} on linear HDR values (shown to correlate well with subjective quality assessment [21]). Baselines are run with the authors’ official code and, where available, official pretrained weights; we only convert our data to their input format. As in the evaluation of Rainer et al. [34], the evaluated materials are part of every method’s training set.

Implementation. All models are trained in PyTorch with Adam on a single NVIDIA A100; the compact model uses bfloat16 mixed precision, with the Fourier features in float32. Each per-material network is trained for 30k steps with batches of 32,768 queries and a learning rate of 5×10−45 \times10^{-4}.

MethodFloats/mat.PSNR↑SSIM↑LPIPS↓ΔE\Delta E↓
Per-material fitting
GGX, per-texel anisotropic1.76M22.100.7030.23424.80
NeuMIP [22]2.62M28.340.7820.19211.19
Xia et al. [47]3.32M33.240.8980.0846.53
Ours, per-material3.42M36.440.9280.0604.54
Shared model across materials
Neural Biplanes [10]2.88M26.160.6390.35515.65
Rainer et al., d=8d=8 [34]1.60M29.900.7920.1719.77
Rainer et al., d=32d=32 [34]5.44M31.370.8620.1328.67
Ours, compact25632.560.8200.1836.77

caption: Reconstruction on nine UBO2014 materials. Floats/mat.: numbers stored per material, excluding decoders shared by all materials. Even counting shared decoders, amortised over the 84 materials, the compact model stores 4–14× fewer values than Rainer et al. (≈385k vs. 1.6M and 5.44M). Rainer et al. store per texel a dd-dimensional latent plus the mean and standard deviation used to normalise its values.

:::

The compact model is trained for 705k steps with a learning rate of at most 5×10−45 \times10^{-4} and cosine decay. The flow is trained for 8k steps. Further training details are in supplementary Sec. S1.

Reconstruction

Table 1 groups methods by whether a network is fitted to each material or shared across materials.

Per-material fitting. Our per-material network outperforms the recent neural BTF of Xia et al. [47] by 3.2 dB PSNR at matched size (3.42M vs. 3.32M parameters), with better SSIM, LPIPS and colour error on every one of the nine materials (Figure 5). A per-texel anisotropic GGX fit [45], which can express any material produced by metallic–roughness PBR generators, is 14 dB below our network. As a per-texel BRDF, it treats each texel in isolation and cannot represent the parallax, masking and shadowing between neighbouring texels that a BTF captures [5].

Per-material reconstruction

Figure 5. Per-material reconstruction. First column: the full measured image, with the crop shown for every method outlined; same light–view pair and exposure for all methods in a row; PSNR of the full image. The compact model is compared qualitatively in supplementary Sec. S2.

Shared models. Our compact model reaches a higher PSNR and lower colour error than Rainer et al. [34] with 32-dimensional per-texel latents, which shift the colours of the measured materials (supplementary Sec. S2). Even counting its 32.3M-parameter shared decoder, amortised over the 84 materials, it stores about 385k values per material, 14×14 \times fewer than their 5.44M; the per-material part is a 256-float code. Per-texel latents keep a lower LPIPS (0.132 vs. 0.183), but at 14×14 \times the storage per material (Figure 4). The compact model is the only method at the kilobyte scale: a whole measured BTF, about 101010^{10} values, is described by a single 256-float code, with one decoder shared by all materials. Every material thus becomes a single point in a common space from which materials can be sampled (§ 4.2), whereas a per-texel representation remains an image of millions of features per material. Qualitative comparisons of the compact model are in supplementary Sec. S2.

Figure 4. Rate–distortion on nine UBO2014 materials: PSNR against the number of values stored per material (log scale). The compact model is shown with its 256-float code and with the shared decoder amortised over 84 materials.

Generation

Baselines and setup. No previous generator learns from measured BTFs; the closest material generators learn from flash photographs [13, 61], which can be captured in large numbers, and are therefore the natural image-based baselines for our flow. As they cannot be compared with ours in BTF space, we compare all generators as flash photographs: the training materials are rendered from their BTFs with a point light at the camera, in the capture geometry of Henzler et al. (field of view 45∘45^\circ, 384×384384 \times384 pixels, gamma 2.2); these 886 images are the training data of the image-based baselines and the reference for all metrics. Our samples and those of the Gaussian baselines are rendered by our decoder in the same geometry. To test whether our code space needs a flow-based density estimator, we compare with established density models over the same codes, which also share our decoder: a per-class Gaussian with full covariance fitted to the flow’s training codes, and the same Gaussian restricted to the top 16 principal directions. We retrain Henzler et al. [13] with their official code and default settings (15 epochs) on the 886 training images. PhotoMat [61] is trained on thousands of real flash photographs; we finetune their released 256×256256 \times256 generator on our 886 images with their training code. We render its samples with the light at the camera.

Qualitative comparison. Figure 6 shows random samples of every generator, without selection. Ours and the Gaussian baselines are sampled for each class directly. Neither image-based generator is trained with classes: Henzler et al. encode each training image, so we sample each class from a Gaussian over the codes of its images, while PhotoMat has no encoder, so each of its samples is assigned the class of its nearest training image in DINOv2 features. Our samples keep the structures of the measured materials of their class: the loops of carpet, the grids and twills of fabric, the cells of leather, the grain of wood. Henzler et al. give sharp textures with more variety within a class and the highest coverage, as their encoder builds on an ImageNetpretrained network, but as analytic SVBRDFs they cannot represent the self-shadowing and parallax of measured BTFs. PhotoMat, finetuned on our 886 images, produces realistic wood, leather, felt and stone in places, but collapses to a few modes and breaks down elsewhere: wood-like planks appear in the wallpaper and fabric rows, and several fabric and leather samples are broken, high-contrast patches. The Gaussian baselines sample everywhere around the class mean, but most of their samples land between materials: decoded, they are blurry or show the structure of another class, such as fabric weaves in stone.

Generated materials as flash photographs

Figure 6. Generated materials as flash photographs: four random samples per class of each generator, without selection; shown is the central half of each photograph, the region used for KID. The Gaussian baselines sample our code space and are decoded by our decoder. Henzler et al. are retrained on our 886 images and sample each class from a Gaussian over the latent codes of its training images. PhotoMat is finetuned from its released model and is unconditional, so we group its samples by the class of their nearest training image in DINOv2 features.

Figure 7 shows this in our code space: Gaussian and PCA samples fall between the classes and mix their characteristics, while ours remain within their own class and share its characteristics. Because every sample is a full BTF, it can be relit (Figure 8 and supplementary Sec. S6: highlights, shadows and relief follow the light as in the measured data). Rendering a sample needs a one-off decoding into a light–view table (3.7 min on one A100; supplementary Sec. S7); after that, it costs the same as a tabulated measured BTF.

Relighting generated materials

Figure 8. Relighting generated materials. Each column is one generated sample, rendered with a fixed camera and a point light at the camera (top row) and moved left, up, right and down (rows 2–5). Exposure is fixed per column.

Quantitative comparison. We draw 96 samples per class (672 in total) and compare them with the 886 training images in DINOv2 features [29, 38]: KID [2] measures the distance between the two distributions, MMD how closely the samples match the training images, and coverage (COV) their diversity [1]. All images are compared on their central half, at the scale of PhotoMat’s samples (supplementary Sec. S4). Our flow has the lowest KID and MMD (Table 2), and decoded training materials, the best our codes allow, reach a KID of 372.

GeneratorKID↓\downarrowMMD↓\downarrowCOV↑\uparrow
Gaussian1222 ± 610.27618.6
PCA-Gaussian1160 ± 400.26521.1
Henzler et al. [13]698 ± 200.25223.0
PhotoMat [61]863 ± 290.34120.2
Ours, 84 measured656 ± 440.24210.8
Ours, with colourways466 ± 370.215<u>22.1</u>

Table 2. Generation on UBO2014: 672 samples per generator as flash photographs against the flash photographs of the 886 training materials, in DINOv2 features. KID×103\times10^{3}, ± s.d. over ten random halves; COV in %. Best in bold, second best underlined. All generators are trained on the same 886 materials, except the first row of ours; PhotoMat is finetuned from its released model.

Ablations

Colourways. Trained on the 84 measured materials only, the same flow copies a few training materials (Figure 9). With colourways, KID improves from 656 to 466 and coverage doubles from 10.8% to 22.1% (Table 2): the generator covers many more colours of the measured structures.

Colourways ablation

Figure 9. Colourways ablation: eight random samples per class of the flow trained on the 84 measured materials only (original) and with colourways (augmented), as flash photographs. Without colourways the flow repeats a few training materials.

Discussion

Global codes versus per-texel latents. Per-texel latent maps [34, 47] and feature planes [10] store most of a material’s information explicitly and use a small decoder; our compact representation stores almost none and uses a large shared decoder. For reconstruction this trades perceptual sharpness for far smaller per-material storage. For generation, a per-texel representation is an image of 8–32 features per texel, 1.6–5.4M values per material, and generating one requires an image-scale generative model. A global code reduces each material to a point in a 256-dimensional space, where a small flow models their distribution.

Generated codes: t-SNE plots of training and generated material codes

Figure 7. Generated codes. t-SNE of the training codes (measured materials and colourways) and of the 672 samples of each class-conditional generator in our code space, coloured by the class they were generated for. The first panel shows the training codes by class; in the others they are shown in grey. Thumbnails show one random sample per class, rendered on a test object. Henzler et al. and PhotoMat are not included, as they do not generate codes in our space.

Extent and tiling. Our decoder covers the measured patch. Xia et al. [47] extend a single material to larger extents with a diffusion model over its latent textures, which is complementary to our generation and could extend it.

Limitations. With twelve measured materials per class, our generator learns the structures it has seen: its samples stay close to the measured materials and their colourways, and structural novelty is limited to a few blends of two training materials (supplementary Sec. S5). Also, rendering requires decoding each material once into a light–view table.

Conclusion

We presented neural representations of measured BTFs at two operating points: a per-material network that represents a measured BTF accurately, and a compact

model that reduces each material to a single code. In this code space, measured BTFs become sampleable: a class-conditional flow, trained on the measured materials enlarged with colourways and hue rotations, generates materials that render and relight as full BTFs. This is a first step towards bringing the realism of measured appearance to generated materials in rendering pipelines. Larger captures such as RoboCloth [24], augmentation that varies the structure of materials as well as their colour, and conditioning on text prompts or images instead of classes are natural next steps. We hope this work makes measured appearance a practical target for generative modelling.

References

  1. [1]Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning representations and generative models for 3D point clouds. In International Conference on Machine Learning (ICML), pages 40–49, 2018. 7arxiv.org/abs/1707.02392
  2. [2]Mikołaj Bińkowski, Danica J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In International Conference on Learning Representations (ICLR), 2018. 7arxiv.org/abs/1801.01401
  3. [3]Brent Burley. Physically-based shading at Disney. In ACM SIGGRAPH Courses, 2012. 2
  4. [4]Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramèr, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. In USENIX Security Symposium, 2023. 3
  5. [5]Kristin J. Dana, Bram van Ginneken, Shree K. Nayar, and Jan J. Koenderink. Reflectance and texture of real-world surfaces. ACM Transactions on Graphics, 18(1):1–34, 1999. 1, 2, 6
  6. [6]Stavros Diolatzis, Jan Novák, Fabrice Rousselle, Jonathan Granskog, Miika Aittala, et al. Mesogan: Generative neural reflectance shells. Computer Graphics Forum, 2023. 3DOI
  7. [7]Yishun Dou, Zhong Zheng, Qiaoqiao Jin, Bingbing Ni, Yu-gang Chen, and Junxiang Ke. Real-time neural BRDF with spherically distributed primitives. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4337–4346, 2024. 2arxiv.org/abs/2310.08332
  8. [8]Emilien Dupont, Hyunjik Kim, S. M. Ali Eslami, Danilo Jimenez Rezende, and Dan Rosenbaum. From data to functa: Your data point is a function and you can treat it like one. In ICML, 2022. 2arxiv.org/abs/2201.12204
  9. [9]Ziya Erkoç, Fangchang Ma, Qi Shan, Matthias Nießner, and Angela Dai. Hyperdiffusion: Generating implicit neural fields with weight-space diffusion. In ICCV, pages 14300–14310, 2023. 2arxiv.org/abs/2303.17015
  10. [10]Jiahui Fan, Beibei Wang, Miloš Hašan, Jian Yang, and Ling-Qi Yan. Neural biplane representation for BTF rendering and acquisition. In ACM SIGGRAPH Conference Proceedings, 2023. 2, 5, 7DOI
  11. [11]Fazilet Gokbudak, Alejandro Sztrajman, Chenliang Zhou, Fangcheng Zhong, Rafal Mantiuk, and Cengiz Oztireli. Hypernetworks for generalizable BRDF representation. In European Conference on Computer Vision (ECCV), 2024. 2arxiv.org/abs/2311.15783
  12. [12]Yu Guo, Cameron Smith, Miloš Hašan, Kalyan Sunkavalli, and Shuang Zhao. Materialgan: Reflectance capture using a generative svbrdf model. ACM Transactions on Graphics, 39(6), 2020. 3DOI
  13. [13]Philipp Henzler, Valentin Deschaintre, Niloy J. Mitra, and Tobias Ritschel. Generative modeling of BRDF textures from flash images. ACM Transactions on Graphics, 40(6), 2021. 2, 3, 6, 7arxiv.org/abs/2102.11861
  14. [14]Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022. 5arxiv.org/abs/2207.12598
  15. [15]Bingyang Hu, Jie Guo, Yanjun Chen, Mengtian Li, and Yanwen Guo. DeepBRDF: A deep representation for manipulating measured BRDF. Computer Graphics Forum, 39(2), 2020. 2DOI
  16. [16]Jay Idema and Pieter Peers. Neural appearance modeling from single images. arXiv preprint arXiv:2406.18593, 2024. 2arxiv.org/abs/2406.18593
  17. [17]Wenzel Jakob, Sébastien Speierer, Nicolas Roussel, Merlin Nimier-David, Delio Vicini, Tizian Zeltner, Baptiste Nicolet, Miguel Crespo, Vincent Leroy, and Ziyi Zhang. Mitsuba 3 renderer, 2022. https://mitsuba-renderer.org. 1
  18. [18]Brian Karis. Real shading in Unreal Engine 4. In ACM SIGGRAPH Course: Physically Based Shading in Theory and Practice, 2013. 2
  19. [19]Behnaz Kavoosighafi, Jeppe Revall Frisvad, Saghi Hajisharif, Jonas Unger, and Ehsan Miandji. Sparsebtf: Sparse representation learning for bidirectional texture functions. In Eurographics Symposium on Rendering, 2023. 2DOI
  20. [20]Behnaz Kavoosighafi, Saghi Hajisharif, Ehsan Miandji, et al. Deep svbrdf acquisition and modelling: A survey. Computer Graphics Forum, 2024. 3DOI
  21. [21]Behnaz Kavoosighafi, Rafal K Mantiuk, Saghi Hajisharif, Ehsan Miandji, and Jonas Unger. Perceived quality of BRDF models. Computer Graphics Forum (Proc. of EGSR), 2025. 5DOI
  22. [22]Alexandr Kuznetsov, Krishna Mullie, Zexiang Xu, Miloš Hašan, and Ravi Ramamoorthi. NeuMIP: Multi-resolution neural materials. ACM Transactions on Graphics, 40(4), 2021. 2, 5
  23. [23]Zhiqiang Li, Xu-Kun Shen, Xueyang Zhou, Yong Hu, and Bowen Li. Hyper-snbrdf: Hypernetwork for neural brdf using sinusoidal activation. In International Conference on 3D Vision (3DV), 2024. 2DOI
  24. [24]Zhen Li, Haoran Qin, Francesco Di Sario, Daniel Rebain, and Andrea Tagliasacchi. Roblocloth: A large-scale real cloth material dataset for neural reflectance reconstruction. arXiv preprint arXiv:2610.21090, 2026. 2, 3, 8arxiv.org/abs/2610.21090
  25. [25]Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. In International Conference on Learning Representations (ICLR), 2023. 2, 4arxiv.org/abs/2210.02747
  26. [26]Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In International Conference on Learning Representations (ICLR), 2023. 2, 4arxiv.org/abs/2209.03003
  27. [27]Ivan Lopes, Fabio Pizzati, and Raoul de Charette. Material palette: Extraction of materials from a single image. In CVPR, pages 4379–4388, 2024. 3arxiv.org/abs/2311.17060
  28. [28]Jacob Munkberg, Peter Hedman, Francis Keller, and Jon Hasselgren. Texture space material diffusion. arXiv preprint arXiv:2609.37654, 2026. 2, 3arxiv.org/abs/2609.37654
  29. [29]Maxime Oquab, Timothée Darcet, Théo Moutakanni, et al. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. 7arxiv.org/abs/2304.07193
  30. [30]Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. DeepSDF: Learning continuous signed distance functions for shape representation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 2arxiv.org/abs/1901.05103
  31. [31]François Pitié and Anil Kokaram. The linear monge-kantorovitch linear colour mapping for example-based colour transfer. In European Conference on Visual Media Production (CVMP), 2007. 3, 4
  32. [32]Nithin Raghavan, Krishna Mullia, Alexander Trevithick, Fuzjun Luan, Miloš Hašan, and Ravi Ramamoorthi. Generative neural materials. In ACM SIGGRAPH Conference Proceedings, 2025. 2, 3
  33. [33]Gilles Rainer, Wenzel Jakob, Abhijeet Ghosh, and Tim Weyrich. Neural BTF compression and interpolation. Computer Graphics Forum, 38(2):235–244, 2019. 2DOI
  34. [34]Gilles Rainer, Abhijeet Ghosh, Wenzel Jakob, and Tim Weyrich. Unified neural encoding of BTFs. Computer Graphics Forum, 39(2), 2020. 2, 5, 6, 7DOI
  35. [35]Carlos Rodriguez-Pardo, Konstantinos Kazatzis, Jorge Lopez-Moreno, and Elena Garces. Neubtf: Neural fields for btf encoding and transfer. Computers & Graphics, 114:239–246, 2023. 2
  36. [36]Szymon M. Rusinkiewicz. A new change of variables for efficient BRDF representation. In Eurographics Workshop on Rendering, 1998. 3DOI
  37. [37]Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffusion art or digital forgery? investigating data replication in diffusion models. In CVPR, 2023. 3
  38. [38]George Stein, Jesse C. Cresswell, Rasa Hosseinzadeh, et al. Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models. In NeurIPS, 2023. 7arxiv.org/abs/2306.04675
  39. [39]Alejandro Sztrajman, Gilles Rainer, Tobias Ritschel, and Tim Weyrich. Neural BRDF representation and importance sampling. Computer Graphics Forum, 40(6), 2021. 2, 3arxiv.org/abs/2102.05963
  40. [40]Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In Advances in Neural Information Processing Systems (NeurIPS), 2020. 2, 3arxiv.org/abs/2006.10739
  41. [41]Giuseppe Vecchio. Stablematerials: Enhancing diversity in material generation via semi-supervised learning. arXiv preprint arXiv:2406.09293, 2024. 3arxiv.org/abs/2406.09293
  42. [42]Giuseppe Vecchio and Valentin Deschaintre. Matsynth: A modern pbr materials dataset. In CVPR, pages 22109–22118, 2024. 3DOI
  43. [43]Giuseppe Vecchio, Rosalie Martin, Arthur Roullier, Adrien Kaiser, Romain Rouffet, Valentin Deschaintre, and Tamy Boubekeur. Controlmat: A controlled generative approach to material capture. ACM Transactions on Graphics, 2024. 3DOI
  44. [44]Giuseppe Vecchio, Renato Sortino, Simone Palazzo, and Concetto Spampinato. Matfuse: Controllable material generation with diffusion models. In CVPR, 2024. 3DOI
  45. [45]Bruce Walter, Stephen R. Marschner, Hongsong Li, and Kenneth E. Torrance. Microfacet models for refraction through rough surfaces. In Eurographics Symposium on Rendering, 2007. 5DOI
  46. [46]Michael Weinmann, Juergen Gall, and Reinhard Klein. Material classification based on training data synthesized using a BTF database. In European Conference on Computer Vision (ECCV), 2014. 2, 5DOI
  47. [47]Mengqi Xia, Yuzhen Zhang, Yutong Yi, Sumit Chaturvedi, Emilie Nogué, Yiwei Hu, Julie Dorsey, and Wenzel Jakob. Neural representation and latent space synthesis of bidirectional texture functions. In Eurographics Symposium on Rendering, 2026. 2, 5, 7, 8DOI
  48. [48]Zilin Xu, Zahra Montazeri, Beibei Wang, and Ling-Qi Yan. A dynamic by-example btf synthesis scheme. In SIGGRAPH Asia 2024 Conference Papers, 2024. 2arxiv.org/abs/2405.14025
  49. [49]Zilin Xu et al. Towards comprehensive neural materials: Dynamic structure-preserving synthesis with accurate silhouette at instant inference speed. In ACM SIGGRAPH 2025 Conference Papers, pages 161:1–161:11, 2025. 2DOI
  50. [50]Bowen Xue, Shuang Zhao, Henrik Wann Jensen, and Zahra Montazeri. A hierarchical architecture for neural materials. Computer Graphics Forum, 43(6), 2024. 2arxiv.org/abs/2307.10135
  51. [51]Bowen Xue, Saeed Hadadan, Zheng Zeng, Fabrice Rousselle, Zahra Montazeri, and Miloš Hašan. Videoneumat: Neural material extraction from generative video models. In ACM SIGGRAPH 2026 Conference Papers, 2026. 3
  52. [52]Kim Youwang, Jon Hasselgren, Peter Kocsis, Andrea Weidlich, Tae-Hyun Oh, and Jacob Munkberg. Extracting neural materials from images. arXiv preprint arXiv:2606.26715, 2026. 3arxiv.org/abs/2606.26715
  53. [53]Yunchen Yu, Jacob Munkberg, Jon Hasselgren, Chris Cummings, Steve Marschner, and Andrea Weidlich. Toward richer material generation via procedural data enhancement. In ACM SIGGRAPH 2026 Conference Papers, 2026. 3
  54. [54]Manzil Zaheer, Satwik Kottur, Siamak Ravankhahsh, Barnabas Poczos, Ruslan Salakhutdinov, and Alexander Smola. Deep sets. In Advances in Neural Information Processing Systems (NeurIPS), 2017. 3
  55. [55]Tizian Zeltner, Fabrice Rousselle, et al. Real-time neural appearance models. ACM Transactions on Graphics, 43(3):33:1–33:17, 2024. 2DOI
  56. [56]Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 5arxiv.org/abs/1801.03924
  57. [57]Xiaohua Zhang, Junyu Dong, Yanhai Gan, Hui Yu, and Lin Qi. BTF data generation based on deep learning. Procedia Computer Science, 147:233–239, 2019. 3DOI
  58. [58]Chenliang Zhou, Zheyuan Hu, Alejandro Sztrajman, Yancheng Cai, Yaru Liu, and Cengiz Oztireli. M³sh: Multi-modal material synthesis via hyperdiffusion. In AAAI Conference on Artificial Intelligence, 2026. 3
  59. [59]C. Zhou, Alejandro Sztrajman, Gilles Rainer, Fangcheng Zhong, Fazilet Gobukad, et al. Physically based neural brdf: A framework for physically correct material reconstruction, generation and editing. Computer Graphics Forum, 2026. 2
  60. [60]Xilong Zhou, Miloš Hašan, Valentin Deschaintre, Paul Guerrero, Kalyan Sunkavalli, and Nima Khademi Kalantari. Tile-gen: Tileable, controllable material generation and capture. In SIGGRAPH Asia 2022 Conference Papers, 2022. 3
  61. [61]Xilong Zhou, Miloš Hašan, Valentin Deschaintre, Paul Guerrero, Yannick Hold-Geoffroy, Kalyan Sunkavalli, and Nima Khademi Kalantari. Photomat: A material generator learned from single flash photos. In ACM SIGGRAPH 2023 Conference Papers, 2023. 2, 3, 6, 7
  62. [62]Xilong Zhou, Pedro Figueiredo, Miloš Hašan, Valentin Deschaintre, Paul Guerrero, Yiwei Hu, and Nima Khademi Kalantari. Realmat: Realistic materials with diffusion and reinforcement learning. arXiv preprint arXiv:2509.01134, 2025. 3DOI

Paper details

vlnc:W_dnrdjyt8
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents