WorksComputer Science
Choosing the Right Representations: Task-Aware Fusion for Multimodal Materials Language Models
Abstract
Multimodal language models for materials science rely on effective mechanisms for incorporating three-dimensional structural representations. MatterChat addresses this challenge using a large bridge module between a pretrained structural encoder and the LLM, together with a separate contrastive alignment stage. We instead ask whether this bridge can adaptively select and combine representations from multiple structural encoders according to the target property. We find that supervised and generative material encoders provide complementary information for property prediction. We found that, for Zatom-1, this information is not concentrated in the final layer. Instead, the most informative representations occur at different depths depending on the target property. This motivates Task-Conditioned Attention Fusion (TCAF), a lightweight attention-based bridge that uses task-conditioned latent queries to combine CHGNet features with intermediate Zatom-1 representations. The bridge is trained directly on downstream prediction tasks, allowing representation selection and material-language alignment to be learned jointly without separate contrastive pretraining. Across classification and regression tasks, TCAF outperforms MatterChat, with up to 47.8% lower RMSE, while using substantially fewer trainable parameters and halving the overall training time. Further experiments incorporating UMA-S demonstrate TCAF's flexibility across encoder combinations, with the three-encoder configuration achieving the highest macro-average classification accuracy of 89.61%.
Introduction
In silico materials discovery has traditionally relied on high-fidelity computational methods such as density functional theory [1] and ab initio molecular dynamics [2]. While accurate, these methods are computationally expensive and difficult to scale across large chemical spaces [3]. Graph-based machine-learning interatomic potentials (MLIPs) [4, 5, 6] address this limitation by learning representations of local atomic environments [7] and have substantially accelerated material-property prediction. However, these models operate on atomic structures alone and cannot directly incorporate natural-language queries or scientific context [8].
Large language models (LLMs) offer a complementary way to interact with materials data through natural language [9, 10]. Most existing approaches represent materials using one-dimensional text formats, including chemical formulas [9], SMILES strings [10, 11, 12], textual descriptions [13], and crystallographic information files [14]. These representations can encode structural information, but processing them as token sequences does not provide the geometric inductive biases used by models designed explicitly for three-dimensional atomic environments. As a result, text-based models generally remain less accurate than dedicated graph models for quantitative property prediction [13]. This motivates multimodal models that connect 3D material structures with LLMs.
MatterChat [3] takes an important step in this direction by combining material graphs with natural-language queries. It uses CHGNet [15] as a pretrained structural encoder and connects its atom-level representations to an LLM through a BLIP-2-style bridge [16]. The bridge consists of alternating self- and cross-attention layers and is trained in two stages: contrastive pretraining first aligns the structural and textual modalities, followed by supervised fine-tuning for downstream property prediction. While effective, this design introduces a large number of trainable parameters and requires a separate alignment stage. It also relies on a single structural encoder, although different encoders may provide complementary information for different material properties.
This raises a more general question: rather than committing to a single structural representation, can the bridge adaptively select and combine representations according to the target property? We investigate this question using two structurally different encoders: CHGNet, a supervised MLIP, and Zatom-1 [17], a general-purpose multimodal flow-matching model [18] for three-dimensional molecules and materials. Our analysis shows that the two encoders provide complementary information for property prediction. We also find that, for Zatom-1, useful information is not concentrated in the final layer. Different intermediate layers are most informative for different material properties, suggesting that a fixed representation from the generative encoder is unnecessarily restrictive.
These observations motivate Task-Conditioned Attention Fusion (TCAF). TCAF uses a lightweight attention-based bridge model based on Perceiver [19, 20] architecture to combine representations from CHGNet with intermediate representations from Zatom-1. The bridge operates through cross-attention [21], with latent queries conditioned on the target property using a mechanism inspired by prompt tuning [22]. These task-conditioned queries allow the bridge to change which encoder and which Zatom-1 layers it attends to for different material-property queries. Thus, instead of aligning each structural representation with language in advance, the bridge learns representation selection, fusion, and material-language alignment directly from the downstream training objective. Interestingly, this formulation also removes the need for a separate contrastive pretraining stage. The resulting model is trained in a single stage while using fewer trainable parameters than MatterChat. Across classification and regression tasks, TCAF improves predictive performance over MatterChat and reduces the overall training time by half. We further evaluate TCAF with encoder combinations incorporating UMA-S [23], demonstrating its flexibility in integrating additional structural representations.
The paper presents three key contributions:
Task-conditioned representation fusion. We introduce TCAF, a single-stage framework with a lighter bridge model than the heavy two-stage MatterChat. TCAF additionally replaces static structural alignment with task-conditioned latent queries, allowing the model to dynamically route attention across supervised features, leading to a consistent improvement on all tasks.
Analysis of generative structural representations. We show that property-relevant information in Zatom-1 is distributed across depth and that the most informative layers vary by material property. We further show that these representations complement those learned by CHGNet.
Empirical performance and efficiency. TCAF achieves the highest classification accuracy across all evaluated tasks and the lowest RMSE on all three regression tasks, reducing regression RMSE by up to 47.8% while using at least 80% fewer trainable parameters and half the training time.
Method: Task-conditioned Representation Fusion (TCAF)
The proposed framework combines 3D crystal structures with natural-language queries for material-property prediction, as illustrated in Figure 1. The input structure is processed by two frozen structural encoders: Zatom-1, a generative model, and CHGNet, a supervised machine-learning interatomic potential. CHGNet produces 64-dimensional atom embeddings, which we project to 512 dimensions to match the Zatom-1 representations. Representations from both encoders are passed to a trainable Perceiver bridge model, which combines them through cross-attention. The resulting latent tokens are appended to the natural-language prompt and passed to a frozen LLM.

Figure 1. Overview of our proposed Task-Conditioned Attention Fusion (TCAF) methodology. (a) Frozen domain encoders (Zatom-1 and CHGNet) extract complementary atom-level representations from a crystal structure. Here, features from all 16 Zatom-1 layers are retained, and CHGNet features are projected from 64 to 512 dimensions. (b) The property requested by the prompt selects a learned task residual, which is added to 32 latent queries. Two Perceiver blocks apply cross-attention to the structural features, followed by latent self-attention and a feed-forward network. These task-conditioned queries allow the bridge to weight encoder sources and layer depths differently for each property. (c) The fused latents are projected into the language-model embedding space and combined with prompt embeddings. A frozen language model then generates the response. The trainable components are optimized jointly using a downstream language-modeling objective.
Single-Stage Training. MatterChat uses a parameter-heavy BLIP-2-style bridge and a separate contrastive pretraining stage to align structural and textual representations. We instead use a lightweight attention-based bridge that compresses the variable-length sequence of atom embeddings into 32 latent tokens. Its cross-attention is mask-aware, ensuring that padded atoms do not contribute to the resampling operation. The bridge is trained directly through the language-modeling objective of the frozen LLM on the downstream prediction tasks. This allows material-language alignment to be learned together with the task objective and removes the need for separate contrastive pretraining. As a result, the complete model is trained in a single stage, reducing the overall training time by half.
Task-Conditioned Attention Fusion. Our analysis shows that property-relevant information in Zatom-1 is distributed across network depth rather than concentrated in its final layer (see Table 1). Therefore, we aim to design a method that can pass the representations from multiple Zatom-1 layers to the Perceiver together with the projected CHGNet representation. These representations form the keys and values of the cross-attention operation, while 32 learned latent tokens act as queries. Each Perceiver block applies cross-attention followed by latent self-attention and a feed-forward network [21]. To make this fusion depend on the target property, we "dynamically modify" the latent queries using a mechanism inspired by prompt tuning [22]. For each material property, we learn a query residual that is added to the initial latent tokens before the attention blocks. The property specified by the natural-language prompt determines which residual is used. The resulting task-conditioned queries allow the bridge to adapt its attention to both encoder source and Zatom-1 depth. While we focus on Zatom-1 and CHGNet in the primary configuration, TCAF is not restricted to this encoder pair. We additionally evaluate encoder combinations incorporating UMA-S to assess the flexibility of the fusion framework, as reported in § 3.
| Task | 1st Best | 2nd Best | 3rd Best |
| Metallic | L0 | L1 | L2 |
| Direct Bandgap | L2 | L7 | L10 |
| Stability | L0 | L2 | L6 |
| Exp Ob | L1 | L2 | L0 |
| Is Magnetic | L0 | L1 | L2 |
| Magnetic Order | L0 | L2 | L1 |
Table 1. Top-3 best-performing Zatom-1 layers for each classification task, identified via linear probing.
Experimental Analysis
We evaluate our framework on the materials science instruction-tuning dataset introduced with MatterChat [3]. We use a frozen Qwen2.5-1.5B language model across all experiments. We compare TCAF with MatterChat and a Projector baseline, which uses an MLP-based bridge to map structural representations into 32 latent tokens without attention.
Identifying Property-Discriminative Layers. We first investigate which layers of the frozen Zatom-1 encoder contain the most property-relevant information. Because the final layers are optimized for its pretraining objective, intermediate representations may be better suited for downstream property prediction. We test this by training a logistic regression probe on mean-pooled representations from each of the 16 Zatom-1 layers. As shown in Table 1, property-relevant information is concentrated in the earlier layers of Zatom-1. Layer 0 (L0) performs best for most classification tasks, while Layer 2 (L2) ranks among the top three layers across all evaluated properties. We next evaluate L0 and L2 in the full LLM pipeline and compare them with the final Zatom-1 layer (L15) and the supervised CHGNet baseline. As shown in Table 2, L15 consistently performs worst, indicating that the final layer of the generative encoder might not have the most useful representation for material-property prediction. L2, in contrast, is competitive with CHGNet across several tasks. However, no single Zatom-1 layer consistently outperforms CHGNet across all properties. These results show that the two encoders provide complementary representations and motivate their adaptive fusion.
| Task | L0 | L2 | L15 | CHGNet |
| Metallic [↑] | 0.7758 | 0.8331 | 0.8218 | 0.6986 |
| Direct Bandgap [↑] | 0.8424 | 0.8751 | 0.8708 | 0.8698 |
| Stability [↑] | 0.8274 | 0.8322 | 0.7620 | 0.8695 |
| Exp Ob [↑] | 0.8064 | 0.8171 | 0.7796 | 0.8112 |
| Is Magnetic [↑] | 0.8727 | 0.8596 | 0.8194 | 0.9218 |
| Magnetic Order [↑] | 0.8144 | 0.8170 | 0.8004 | 0.8466 |
Table 2. Classification accuracy comparison between individual Zatom-1 layers and the supervised CHGNet baseline. Bold values indicate the best performance for each task.
Task-Conditioned Attention Fusion Performance. Figure 2 compares the classification accuracy of seven methods across all tasks. TCAF achieves the highest accuracy on each task, showing the benefit of task-conditioned fusion.
Figure 2. Comparison of classification accuracy. Dashed lines indicate average accuracy over tasks.
In contrast, the Projector L2 baseline performs worst overall, suggesting that a simple MLP is insufficient to connect the generative representations with the LLM. Perceiver CHGNet achieves the second-highest average accuracy and performs similarly to MatterChat across most tasks. Unlike MatterChat, however, the Perceiver bridge is trained in a single stage and uses substantially fewer trainable parameters (20M vs. more than 100M of MatterChat). Table 3 reports regression performance together with the number of invalid generations, where an output is considered invalid if the LLM fails to produce a parseable floating-point value. TCAF achieves the lowest RMSE on all three regression tasks (up to 47.8% lower than MatterChat in "Energy Above Hull") while producing only a small number of invalid outputs. This suggests that task-conditioned fusion improves both predictive accuracy and the stability of the generated outputs. Additional experimental details are provided in Appendix A and B, and fusion and task-conditioning ablations, together with cross-attention analyses, are provided in Appendix C.
| Method | Formation Energy | Energy Above Hull | Bandgap | |||
| No. Invalid [] | RMSE [] | No. Invalid [] | RMSE [] | No. Invalid [] | RMSE [] | |
| MatterChat CHGNet | 2 | 0.2944 | 380 | 0.2560 | 1 | 0.6731 |
| MatterChat L2 | 258 | 1.5861 | 13 | 0.3061 | 571 | 1.1968 |
| Projector CHGNet | 62 | 0.2226 | 79 | 0.2795 | 2 | 0.7681 |
| Projector L2 | 594 | 0.6702 | 894 | 0.4713 | 1742 | 1.6955 |
| Perceiver CHGNet | 55 | 0.2095 | 486 | 0.2363 | 46 | 0.9921 |
| Perceiver L2 | 1039 | 0.4896 | 143 | 0.3277 | 94 | 1.2794 |
| TCAF | 1 | 0.1775 | 28 | 0.1336 | 1 | 0.5356 |
Table 3. Regression performance of TCAF versus other methods.
Generalization Across Structural Encoders. To assess the generalizability and modularity of TCAF when incorporating new representations, we integrate UMA-S alongside existing encoders. As summarized in Table 4, applying TCAF across various encoder combinations yields consistent performance gains over baseline strategies. Both MatterChat UMA-S and Perceiver UMA-S achieve lower macro scores of 86.44% and 85.84%. Combining two encoders improves performance: TCAF CHGNet+UMA-S and TCAF Zatom+UMA-S reach macro scores of 88.82% and 89.02%, with the latter achieving the highest accuracy on Direct gap (88.61%) and Magnetic (95.51%). Combining all three encoders (TCAF Zatom+CHGNet+UMA-S) achieves the best overall result, with a top macro score of 89.61%. It secures the highest performance in four out of six tasks. These results show that TCAF can effectively integrate multiple distinct representations to boost overall accuracy.
| Algorithm | Metal | Direct gap | Stable | Observed | Magnetic | Mag. order | Macro |
| [] | [] | [] | [] | [] | [] | [] | |
| Matterchat UMA-S | 87.43 | 85.84 | 86.71 | 79.10 | 92.71 | 86.86 | 86.44 |
| Perceiver UMA-S | 88.53 | 87.35 | 79.02 | 81.22 | 91.83 | 87.09 | 85.84 |
| TCAF CHGNet+UMA-S | 90.81 | 87.50 | 88.14 | 83.16 | 95.17 | 88.14 | 88.82 |
| TCAF Zatom+UMA-S | 86.80 | 88.61 | 88.57 | 86.04 | 95.51 | 88.61 | 89.02 |
| TCAF Zatom+CHGNet+UMA-S | 90.97 | 87.99 | 88.83 | 86.73 | 94.49 | 88.64 | 89.61 |
Table 4. Performance comparison of UMA-S encoder on classification tasks.
Conclusion
We introduced Task-Conditioned Attention Fusion (TCAF), a lightweight framework for multimodal material-property prediction that removes the need for a separate contrastive pretraining stage. Using task-conditioned latent queries within a Perceiver bridge, TCAF dynamically combines intermediate representations from the generative model Zatom-1 with representations from the supervised interatomic potential CHGNet. Our analysis shows that property-relevant information in Zatom-1 is distributed across network depth and varies with the target property, supporting the use of adaptive rather than fixed structural representations. Across classification and regression tasks, TCAF improves predictive performance while using substantially fewer trainable parameters. These results suggest that multimodal material–language models can benefit from selecting and combining complementary structural representations directly during downstream training.
Acknowledgments and Disclosure of Funding
This research used resources of the National Energy Research Scientific Computing Center, a DOE Office of Science User Facility supported by the Office of Science of the U.S. Department of Energy (DOE) under Contract No. DE-AC02-05CH11231, using the AI4Sci@NERSC award NERSC DDRERCAP0036206 awarded to AM. NBE would also like to acknowledge support from the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, through the RAPIDS Institute under the Scientific Discovery through Advanced Scientific Computing (SciDAC) program, under Contract No. DE-AC02-05CH11231 at Lawrence Berkeley National Laboratory. This work was also partially supported by NSF grants CAREER-2401285, RI-2533631, RI-2334936, the LaCross Institute Fellowship and DARPA under Contract No. #HR00112552005. This material is based upon work supported by the National Science Foundation Graduate Research Fellowship Program under Grant No 2234693. The authors also acknowledge the Research Computing at the University of Virginia. The views, opinions, and/or findings expressed are those of the authors and should not be interpreted as representing the official views or policies of the U.S. Government.
Experimental Details
Dataset and Evaluation Protocol
Our experiments utilized the instruction-tuning dataset introduced by the MatterChat [3] framework. The dataset from Matterchat1 encompasses nine distinct material property prediction tasks. Following data cleaning and post-processing, the corpus yields a total of 1,285,144 structure-instruction pairs. The exact distribution of training and test samples across the nine tasks is detailed in Table 5. To address the issue of generative degeneration where language models output unparseable text or have decimal point missing results, we report the number of invalid samples and RSME for the remaining samples.
| Task | Train | Test | Total |
| Is Metal | 128,609 | 14,290 | 142,899 |
| Direct Bandgap | 128,609 | 14,290 | 142,899 |
| Stability | 128,609 | 14,290 | 142,899 |
| Experimentally Observed | 128,609 | 14,290 | 142,899 |
| Is Magnetic | 127,821 | 14,204 | 142,025 |
| Magnetic Order | 128,537 | 14,289 | 142,826 |
| Bandgap | 128,609 | 14,290 | 142,899 |
| Energy Above Hull | 128,609 | 14,290 | 142,899 |
| Formation Energy | 128,609 | 14,290 | 142,899 |
| Total | 1,156,621 | 128,523 | 1,285,144 |
Table 5. Dataset distribution across classification and regression tasks.
Architecture Specifications
The crystal encoders and language model are frozen throughout all experiments. We use Qwen2.5-1.5B-Instruct with a hidden dimension of 1,536. The structural encoders are configured as follows:
CHGNet: A supervised interatomic model producing 64-dimensional per-atom embeddings.
Zatom-1: A 16-layer generative transformer producing 512-dimensional per-atom embeddings at each layer.
UMA-S: A universal interatomic model producing 128-dimensional per-atom embeddings.
We compare three bridge architectures that map structural features into 32 tokens in the language-model embedding space:
Q-Former (MatterChat baseline): Initialized from BERT-base with 12 layers, 12 attention heads, a hidden dimension of 768, a feed-forward width of 3,072, and a dropout rate of 0.1. It uses 32 learned query tokens and applies cross-attention to the structural features every two layers.
MLP Projector: Uses mask-aware adaptive average pooling to obtain 32 structural tokens, followed by a linear projection to 1,536 dimensions, a GELU activation, a second linear projection, and LayerNorm.
Perceiver (TCAF): Uses 32 learned latents of dimension 768, two resampler blocks, 8 attention heads, a feed-forward width of 3,072, and a dropout rate of 0.1. A final projection maps the latents to 1,536 dimensions.
Figure 3 details the computation within TCAF. In our primary Zatom-1 + CHGNet configuration, CHGNet features are projected from 64 to 512 dimensions and combined with representations from all 16 Zatom-1 layers to form the structural memory . Learned 512-dimensional source embeddings distinguish the encoder sources and layer depths. These embeddings are initialized from .
Figure 3. Detailed view of the TCAF bridge. (a) Features from the frozen structural encoders are mapped to a common width of 512 and combined with learned source embeddings to form the structural memory . (b) The prompt determines the task index , which selects one of nine learned residual matrices. The selected residual is added to the shared latent queries, giving . This conditioning occurs once, before the first resampler block. (c) Each block forms queries from the current latents and keys and values from . Query–key similarities determine attention weights over the memory, with padded positions excluded before softmax. The weighted sum retrieves structural information for each query. Outputs from eight heads are concatenated and projected, followed by self-attention across the 32 latent slots and a feed-forward network applied to each slot. (d) Two blocks update the latents while reusing the same structural memory. A final projection produces 32 material tokens of width 1536 for the language model. Batch dimensions are omitted, and the attention equations show one head. Here, is the memory length, and and are the per-head key and value widths.
Task conditioning uses a zero-initialized residual bank of shape , with one residual matrix per prediction task. The task index is inferred exclusively from the prompt text; dataset metadata is not used for task selection. The selected residual is added to the shared learned latents before the first block:
Each task therefore adjusts the full bank of 32 queries. Zero initialization makes the initial queries identical across tasks, while training learns property-specific adjustments.
Within each resampler block, the current latents provide the queries, and the structural memory provides the keys and values. Mask-aware cross-attention retrieves structural information while excluding padded positions. Latent self-attention and a feed-forward network then update the retrieved representations. Both blocks use the same memory , but the second block operates on the updated latents from the first. This lets the requested property influence which encoder features and layer depths the bridge emphasizes. The final 32 material tokens are combined with prompt embeddings and passed to the frozen language model.
Training Hyperparameters
To ensure computational efficiency, all structural embeddings were precomputed and cached on local temporary file systems prior to training. Training was conducted in BF16 mixed precision across 12 distributed hardware accelerator ranks.
Models were trained using the AdamW optimizer with and a weight decay of 0.05. We applied a maximum gradient clipping norm of 1.0. The learning rate followed a linear warm-up schedule from to a peak of over the first 500 optimization steps, subsequently decaying via a cosine schedule down to . With a local batch size of 32 and a gradient accumulation step of 4 across 12 ranks, the effective global batch size was maintained at 1,536. Both pretraining (where applicable) and fine-tuning phases were executed for 7,500 steps. During inference, the maximum generation length was strictly restricted to 256 tokens.
Trainable Parameters
To further highlight the computational advantages of our proposed framework, Table 6 details the number of trainable parameters required by each bridging architecture. The foundational MatterChat framework imposes a substantial computational burden, as its Q-Former bridge necessitates a two-stage training process. During the contrastive pretraining phase (Stage 1), MatterChat requires approximately 197 million to 201 million trainable parameters. Subsequently, during the supervised fine-tuning phase (Stage 2), it still maintains roughly 118 million to 122 million trainable parameters.
| Method | Stage 1 | Stage 2 |
| MatterChat CHGNet | 197,756,477 | 118,020,666 |
| MatterChat L2 | 201,885,245 | 122,149,434 |
| Projector CHGNet | 2,463,744 | |
| Projector L2 | 3,151,872 | |
| Perceiver CHGNet | 20,163,840 | |
| Perceiver L2 | 20,507,904 | |
| TCAF | 20,772,224 |
Table 6. Comparison of trainable parameters across different bridge architectures. For single-stage models (Projector, Perceiver, and TCAF), the training is conducted end-to-end without a separate pretraining phase.
In contrast, the Projector, Perceiver, and our proposed TCAF architectures bypass the resource-intensive pretraining phase, operating exclusively within a single-stage pipeline. While the Projector baseline utilizes the fewest parameters (under 4 million), it severely underperforms in regression and generative stability, as shown in previous sections. TCAF, built upon the Perceiver architecture, requires only 20,772,224 trainable parameters. This effectively demonstrates that TCAF is exceptionally parameter-efficient: it utilizes nearly an order of magnitude fewer trainable parameters compared to MatterChat’s Stage 1 and reduces the Stage 2 parameter count by roughly 80%, all while completely eliminating the multi-stage training paradigm and achieving better predictive performance.
Hardware Infrastructure
All computational workloads, including feature extraction, training, and evaluation, were executed on a large-scale high-performance computing (HPC) cluster. Specifically, the distributed environment utilized a single compute node equipped with 12 advanced hardware accelerator ranks (architecturally equivalent to the Intel Data Center GPU Max Series). Each accelerator rank features 64 GB of High Bandwidth Memory (HBM), providing substantial memory capacity to efficiently handle large-scale multimodal training. Furthermore, these accelerators offer native hardware support for bfloat16 precision, which we leveraged to maximize training throughput and optimize memory bandwidth without the need for gradient scaling.
Data Input-Output and Caching
During distributed training, processing high-resolution structural data often introduces severe I/O bottlenecks. Initially, storing precomputed embeddings as individual files per structure, resulting in over 142,000 tiny files totaling approximately 68 GB. This caused massive read latencies and metadata contention on the shared distributed file system. At a global batch size of 1536, the concurrent read requests from multiple ranks overwhelmed the file system, leading to highly erratic and stalled step times.
To reduce this extreme metadata contention, we implemented an optimized caching pipeline: All per-layer and supervised embeddings were first pre-packed into a single, large contiguous archive. Furthermore, to mitigate I/O bottlenecks and metadata contention typical in distributed file systems, all precomputed embeddings were staged directly to node-local temporary file systems prior to training execution. This node-local staging effectively decoupled the training loop from the shared cluster filesystem, allowing all per-step reads to access high-speed RAM-backed storage. This caching strategy stabilized step times and accelerated overall training throughput by over an order of magnitude, rendering the training process entirely independent of shared file system congestion.
Additional Experiment
To further evaluate the efficacy of the proposed TCAF architecture, we experiment with several fusion variants and ablation settings. Specifically, we implement Concatenation (inspired by [24]) and Weighted Sum (inspired by [25]) as alternative mechanisms to combine the Zatom and CHGNet embeddings while retaining the task-conditioning module. Additionally, we evaluate an ablated version of our model, TCAF (no task cond), to isolate the impact of task-aware latent queries on overall performance.
As illustrated in Figure 4, the fully conditioned TCAF consistently achieves the best performance across all evaluated material property prediction tasks. Conversely, the baseline Matterchat model exhibits the weakest performance across the board, despite utilizing more learnable parameters and requiring longer training times. Among the alternative embedding combination strategies, naive concatenation yields the poorest results. Interestingly, the macro-average performance of the Weighted Sum variant and the unconditioned TCAF are nearly identical (0.8735 and 0.8736, respectively). This plateau suggests that while basic structural fusion techniques can aggregate multi-scale features effectively, dynamic task-specific conditioning is essential to achieve state-of-the-art performance.
Figure 4. Performance comparison of multimodal fusion strategies across six classification tasks. The grouped bars evaluate the baseline Matterchat, basic fusion mechanisms (Weighted Sum, Concatenation), and the proposed TCAF architecture with and without task conditioning. Dashed horizontal lines indicate the macro-average score of each respective method across all tasks, highlighting the overall performance gains achieved by the fully conditioned TCAF.
To understand the mechanism behind TCAF’s performance improvements, we visualize the internal cross-attention weights of the Perceiver blocks to analyze how the model aggregates multimodal features.
In the unconditioned baseline, the absence of task-specific latent queries results in a nearly completely static attention distribution. As demonstrated in Figure 5, the macro-level allocation of attention mass between the aggregated Zatom layers and the CHGNet embedding remains rigid and identical across all nine target properties. This lack of adaptability extends to the micro-level routing; the heatmap in Figure 6 reveals that the unconditioned model uniformly extracts features from the exact same Zatom depth layers regardless of the physical task being evaluated. This behavior indicates an attention collapse, where the model defaults to a generalized, average representation rather than specializing its feature extraction.
Figure 5. Macro-level encoder contribution (Zatom vs. CHGNet) for the baseline Perceiver model without task conditioning. The stacked bars illustrate the mean cross-attention mass allocated to the aggregated 16 Zatom layers compared to the single CHGNet representation across Block 1 and Block 2.
Figure 6. Micro-level cross-attention distribution across the 16 Zatom intermediate layers for the baseline Perceiver model without task conditioning. Color intensity indicates the mean attention mass assigned to each depth layer for different property prediction tasks.
Introducing task conditioning fundamentally alters this behavior, enabling highly dynamic, property-aware feature routing. At the macro-level (Figure 7), the conditioned TCAF actively modulates its reliance on the two distinct structural encoders depending on the query. For thermodynamic properties such as formation_energy and energy_above_hull, the model leans heavily on the CHGNet embedding, allocating a majority of its attention mass (over 50% in Block 2) to this source. Conversely, for quantum-mechanical and electronic targets like is_metal, direct_bandgap, and is_magnetic, the Perceiver shifts its focus dramatically toward the Zatom representation, assigning it nearly 70% of the attention mass in Block 2.
Figure 7. Macro-level encoder contribution (Zatom vs. CHGNet) for the task-conditioned Perceiver model. The stacked bars demonstrate how task-specific latent queries dynamically modulate the balance of cross-attention mass between the aggregated Zatom layers and the CHGNet embedding.
This adaptive routing is further detailed at the micro-level (Figure 8). Rather than applying a uniform extraction pattern, the task-conditioned model learns unique depth-wise selection strategies. heatmap illustrates distinct horizontal variations, proving that the queries dynamically navigate the 16 intermediate Zatom layers to pull specific chemical and structural representations at different abstraction depths. This task-aware modulation confirms that TCAF does not merely aggregate data but actively reasons about which encoder, and which specific layer depth, contains the most relevant physical priors for a given target property.
Figure 8. Micro-level cross-attention distribution across the 16 Zatom intermediate layers for the task-conditioned Perceiver model. The heatmap reveals task-aware depth routing, highlighting how the model adaptively extracts structural and chemical features from different depths based on the target property.
References
- [1]Walter Kohn, Axel D Becke, and Robert G Parr. Density functional theory of electronic structure. The journal of physical chemistry, 100(31):12974–12980, 1996.DOI
- [2]Dominik Marx and Jurg Hutter. Ab initio molecular dynamics: Theory and implementation. Modern methods and algorithms of quantum chemistry, 1(301-449):141, 2000.
- [3]Yingheng Tang, Wenbin Xu, Jie Cao, Weilu Gao, Steven Farrell, Benjamin Erichson, Michael W Mahoney, Andy Nonaka, and Zhi Jackie Yao. A multimodal large language model for materials science. Nature Machine Intelligence, pages 1–14, 2026.DOI
- [4]Simon Batzner, Albert Musaelian, Lixin Sun, Mario Geiger, Jonathan P Mailoa, Mordechai Kornbluth, Nicola Molinari, Tess E Smidt, and Boris Kozinsky. E (3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials. Nature communications, 13(1):2453, 2022.DOI
- [5]Johannes Gasteiger, Florian Becker, and Stephan Günnemann. Gemnet: Universal directional graph neural networks for molecules. Advances in neural information processing systems, 34:6790–6802, 2021.arxiv.org/abs/2106.08903
- [6]Ilyes Batatia, Philipp Benner, Yuan Chiang, Alin M Elena, Dávid P Kovács, Janosh Riebesell, Xavier R Advincula, Mark Asta, Matthew Avaylon, William J Baldwin, et al. A foundation model for atomistic materials chemistry. The Journal of chemical physics, 163(18), 2025.
- [7]Miruna Cretu, Alex Abrudan, Antonia Panescu, Tynan Perez, Rishabh Anand, N Benjamin Erichson, Michael W Mahoney, Samuel Blau, Joseph Jacobson, Rafael Gómez-Bombarelli, et al. Zatom-2: Multitask pretraining on atomistic data for generative modeling across domains. arXiv preprint arXiv:2610.11454, 2026.arxiv.org/abs/2610.11454
- [8]Abeba Birhane, Atoosa Kasirzadeh, David Leslie, and Sandra Wachter. Science in the age of large language models. Nature Reviews Physics, 5(5):277–280, 2023.DOI
- [9]Seongmin Kim, Yousung Jung, and Joshua Schrier. Large language models for inorganic synthesis predictions. Journal of the American Chemical Society, 146(29):19654–19659, 2024.DOI
- [10]Joseph M Cavanagh, Kunyang Sun, Andrew Gritsevskyi, Dorian Bagni, Yingze Wang, Thomas D Bannister, and Teresa Head-Gordon. Smileyllama: Modifying large language models for directed chemical space exploration. Nature Computational Science, pages 1–14, 2026.DOI
- [11]David Weininger. Smiles, a chemical language and information system. 1. introduction to methodology and encoding rules. Journal of chemical information and computer sciences, 28(1):31–36, 1988.DOI
- [12]Huu Binh Ta, Michael Cardei, Alvaro Velasquez, and Ferdinando Fioretto. Search-augmented masked diffusion models for constrained generation. arXiv preprint arXiv:2602.02727, 2026.
- [13]Janghoon Ock, Chakradhar Guntuboinna, and Amir Barati Farimani. Catalyst energy prediction with catberta: unveiling feature exploration strategies through large language models. ACS Catalysis, 13(24):16032–16044, 2023.DOI
- [14]Luis M Antunes, Keith T Butler, and Ricardo Grau-Crespo. Crystal structure generation with autoregressive large language modeling. Nature Communications, 15(1):10570, 2024.DOI
- [15]Bowen Deng, Peichen Zhong, KyuJung Jun, Janosh Riebesell, Kevin Han, Christopher J Bartel, and Gerbrand Ceder. Chgnet as a pretrained universal neural network potential for charge-informed atomistic modelling. Nature Machine Intelligence, 5(9):1031–1041, 2023.DOI
- [16]Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PmLR, 2023.arxiv.org/abs/2301.12597
- [17]Alex Morehead, Miruna Cretu, Antonia Panescu, Rishabh Anand, Maurice Weiler, Tynan Perez, Samuel Blau, Steven Farrell, Wahid Bhimji, Anubhav Jain, et al. Zatom-1: Towards a multimodal foundation model for 3d molecules and materials. arXiv preprint arXiv:2602.22251, 2026.arxiv.org/abs/2602.22251
- [18]Benjamin Kurt Miller, Ricky TQ Chen, Anuroop Sriram, and Brandon M Wood. Flowmm: Generating materials with riemannian flow matching. arXiv preprint arXiv:2406.04713, 2024.arxiv.org/abs/2406.04713
- [19]Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint arXiv:2107.14795, 2021.arxiv.org/abs/2107.14795
- [20]Curtis Hawthorne, Andrew Jaegle, Cătălina Cangea, Sebastian Borgeaud, Charlie Nash, Mateusz Malinowski, Sander Dieleman, Oriol Vinyals, Matthew Botvinick, Ian Simon, et al. General-purpose, long-context autoregressive modeling with perceiver ar. In International Conference on Machine Learning, pages 8355–8558. PMLR, 2022.arxiv.org/abs/2202.07765
- [21]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.arxiv.org/abs/1706.03762
- [22]Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 conference on empirical methods in natural language processing, pages 3045–3059, 2021.arxiv.org/abs/2104.08691
- [23]Brandon Wood, Misko Dzamba, Xiang Fu, Meng Gao, Muhammed Shuaibi, Luis Barroso-Luque, Kareem Abdelmaqsoud, Vahe Gharakhanyan, John Kitchin, Daniel Levine, et al. Uma: A family of universal models for atoms. Advances in Neural Information Processing Systems, 38:129391–129427, 2026.arxiv.org/abs/2506.23971
- [24]Ronghang Hu and Amanpreet Singh. Unit: Multimodal multitask learning with a unified transformer. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 1419–1429. IEEE, 2021.arxiv.org/abs/2102.10772
- [25]Valentin Vielzeuf, Alexis Lechervy, Stéphane Pateux, and Frédéric Jurie. Centralnet: a multilayer approach for multimodal fusion. In European Conference on Computer Vision, pages 575–589. Springer, 2018.arxiv.org/abs/1808.07275






