betaFeedback

WorksPhysics and Astronomy

Discovery and recovery of crystalline materials with property-conditioned transformers

Abstract

Generative models have recently shown great promise for accelerating the design and discovery of new functional materials. Conditional generation enhances this capacity by allowing inverse design, where specific desired properties can be requested during the generation process. However, conditioning of transformer-based approaches, in particular, is constrained by discrete tokenisation schemes and the risk of catastrophic forgetting during fine-tuning. This work introduces CrystaLLM-π (property injection), a conditional autoregressive framework that integrates continuous property representations directly into the transformer's attention mechanism. Two architectures, Property-Key-Value (PKV) Prefix attention and PKV Residual attention, are presented. These methods bypass inefficient sequence-level tokenisation and preserve foundational knowledge from unsupervised pre-training on Crystallographic Information Files (CIFs) as textual input. We establish the efficacy of these mechanisms through systematic robustness studies and evaluate the framework's versatility across two distinct tasks. First, for structure recovery, the model processes high-dimensional, heterogeneous X-ray diffraction patterns, achieving structural accuracy competitive with specialised models and demonstrating applications to experimental structure recovery and polymorph differentiation. Second, for materials discovery, the model is fine-tuned on a specialised photovoltaic dataset to generate novel, stable candidates validated by Density Functional Theory (DFT). It implicitly learns to target optimal band gap regions for high photovoltaic efficiency, demonstrating a capability to map complex structure-property relationships. CrystaLLM-π provides a unified, flexible, and computationally efficient framework for inverse materials design.

Introduction

Developing new functional materials has historically been a time-consuming process. There are many barriers that make the discovery of new materials challenging, including navigating the vast compositional space of inorganic materials and effectively characterising newly synthesised candidates [1, 2, 3, 4]. The combinatorial complexity of materials space, which scales exponentially with the number of elements, coupled with polymorphism and structural complexity, requires intelligent approaches to accelerating discovery [5, 6, 7]. Computational chemistry has helped ameliorate some of these problems by offering accurate predictions of the properties of new materials and complementing modern characterisation techniques [8, 9]. However, accurate physics-based calculations are generally too computationally expensive to allow for wide and fast explorations of materials space.

Generative artificial intelligence has emerged as a powerful tool for materials design [10, 11, 12]. In recent years, two main paradigms for generative materials modelling have emerged: diffusion/flow-based models and transformer-based models [12, 13, 14, 15, 16]. Diffusion and flow approaches have achieved state-of-the-art performance in generating materials both unconditionally and conditioned on specific functional properties [16, 17, 18, 19]. Despite these successes, conditioning these models on new datasets to target specific properties is difficult: it frequently requires important architectural modifications and a two-phase protocol for unconditional pre-training followed by adapter-specific training [20]. Additionally, many diffusion and flow models rely on graph-based representations, where scaling to larger systems is either computationally costly or requires sacrificing long-range dependencies because strong inductive biases, limiting applicability to large and complex chemistries [21].

Transformer-based generative models present an alternative framework for materials design, offering inherent architectural flexibility and favourable scalability by using token-based representations for structure generation [22, 23, 24, 25, 26, 27]. The core selfattention mechanism [28] can contextualise diverse inputs in a unified manner, including compositional, structural and functional property information. Functional property conditioning is currently achieved in transformer-based large language models (LLMs), such as NatureLM and AtomGPT, by fine-tuning general-use models with token-based guidance [29, 30, 31, 32]. However, standard transformer architectures struggle to condition on continuous physical properties effectively. Discrete, digit-level tokenisation inherently disrupts ordinal relationships and prevents smooth mathematical interpolation [33, 34]. Furthermore, using identical discrete tokens to encode continuous global properties and local spatial coordinates or physical descriptors forces a representational conflict within the embedding space. This gap can be resolved with enough data, but is problematic given the relative scarcity of specialised property-labelled crystallographic datasets [35, 36, 37].

This work presents CrystaLLM-π\pi (Property Injection), a transformer-based framework that aims to resolve these limitations. The central hypothesis is that introducing property embeddings as a dedicated attention channel enables targeted material generation while avoiding any disruptive expansion to the sequence length. Two mechanisms are introduced: Property-Key-Value (PKV) Prefix attention and PKV Residual attention. Both methods integrate continuous property information directly into the attention layers, bypassing sequence embedding layers. The mechanisms are designed to preserve the rich structural knowledge from the pre-trained base model, which is based on CrystaLLM [38], while efficiently enabling structure generation steered by functional property targets.

To isolate the impact of these architectural modifications, generative performance is systematically evaluated against standard sequence-based conditioning baselines across datasets ranging from 1K to 653K samples. The generality of the methods is demonstrated across two distinct materials design tasks: the generation of novel stable photovoltaic absorber material candidates with high potential efficiency, verified by Density Functional Theory (DFT), and the recovery of known experimental structures from X-ray diffraction (XRD) data.

The results establish inverse design capabilities of conditional autoregressive crystal structure generation. CrystaLLM-π\pi provides an accessible, lightweight and flexible framework for the recovery and discovery of crystalline materials with a range of targeted functional properties and structural prompts, adaptable to diverse scenarios in materials design without the need for complex architectural redesign or extensive retraining.

Results

CrystaLLM-π Framework

CrystaLLM-π\pi is a transformer-based framework for materials discovery that sequentially generates Crystallographic Information Files (CIFs). The model builds upon the GPT-2 architecture of the original CrystaLLM model [38, 39]. Optimisation focuses on training the model to autoregressively generate chemically viable crystal structures by internalising the syntax and physical constraints inherent to crystallographic data, the mechanisms of which are extensively discussed in the original paper [38].

Integrating continuous functional properties for target-driven inverse design requires key framework modifications compared to unconditional crystal generation. The framework expands the unlabeled pre-training corpus and introduces a condition-aligned batching mechanism (Appendix VI C). A new dynamic boundary tokenisation scheme and a customised format-aware loss function are implemented to stabilise early training (Methods V B, Appendix VI M). To ensure computational scalability and reproducibility, the underlying code adheres to standard Hugging Face tokeniser, model and training protocols, with all datasets and model weights publicly accessible.

Evaluations on the LeMat-Benchmark (Appendix VI E) quantify a set of advantages and limitations of the token-based autoregressive approach. On the non-pre-relaxed MP-20 and Alex-MP-20 benchmarks, CrystaLLM-π\pi achieves the highest structural precision with a root mean square displacement (RMSD) of 0.3619 Å and 0.1914 Å respectively. In the MP-20 section, the model leads competing autoregressive frameworks in primary ranking metrics with a 0.3% Stable, Unique, and Novel (SUN) score and a 3.6% Metastable SUN score. While diffusion architectures currently reach lower convex hull energies, the proposed framework maintains robust structural validity (86.4%) and high out-of-the-box stability, recording the highest overall Metastable rate (48.7%) in this section. CrystaLLM-π\pi however shows a clear restricted novelty rate (24.8%) compared to diffusion models, which often exceed 60%. This constraint suggests a generation distribution biased toward high-density regions of the pre-training manifold, motivating the implementation of a functional property-conditioning framework to actively steer generation toward sparsely seen chemical spaces.

Architectures for Property-Conditioned Generation

While the discrete, digit-wise tokenisation of coordinates facilitates the hierarchical generation of spatial boundaries and physical descriptors, using the same approach for global functional properties introduces substantial architectural inefficiencies. Forcing the model to interpret complex, often multi-dimensional properties through the same discrete token embeddings used for physical descriptors of the material induces a representational conflict and exposes the fundamental numeracy gaps of standard transformers [33, 34].

To resolve these limitations, CrystaLLM-π\pi implements a continuous conditioning framework (Figure 1). During training (Figure 1a), the system tokenises an input CIF into position-encoded embeddings while a dedicated condition encoder simultaneously maps continuous numerical target properties PP directly into an independent, non-linear embedding space. Injecting these continuous representations into every multi-head attention (MHA) layer of the GPT-2 backbone actively biases the generation trajectory, establishing persistent property conditioning with minimal disruption to pre-trained sequence representations in embedding and Feed-Forward Neural Network (FFNN) layers. A cross-entropy loss (Methods V B) is computed between the predicted conditional probability distribution of the next token and the ground-truth sequence. Backpropagation then jointly optimises the parameters of the transformer backbone and the condition encoder.

Image not extracted

Figure 1. Framework for property-conditioned generation of Crystallographic Information Files (CIFs). a. During training, an augmented CIF is converted to token and positional embeddings, while the target property vector PP is mapped by a condition encoder and injected into each multi-head attention layer of the GPT-2 backbone. Next-token predictions are optimised against the shifted ground-truth sequence. b. During inference, the model receives an initial prompt sequence and the target property vector PP, then autoregressively feeds each predicted token back into the model to generate the output CIF. This schematic shows how continuous property information conditions decoding during both training and generation.

During inference (Figure 1b), the model receives a target property vector and an initial prompt sequence. This architecture supports ab initio generation or generation constrained by specific cell compositions and space groups in a unified manner. The model autoregressively samples subsequent tokens to produce a complete CIF, integrating both sequence-level and property-level conditions.

To fairly investigate how best to condition transformer outputs with functional properties, we implemented and benchmarked four distinct architectures. These methods were all designed to integrate a numerical condition vector, c∈RP\mathbf{c} \in\mathbb{R}^{P}, where PP is the number of properties, to guide the generative process. They are grouped into two classes: sequence-level methods and tailored attention mechanisms developed for this work.

Preserving the extensive structural knowledge from unsupervised pre-training is critical in conditional fine-tuning to mitigate catastrophic forgetting [40]. For this goal, a dual optimisation strategy was developed and applied, as detailed in Methods V C. A conservative learning rate was applied to the pre-trained backbone parameters to favour retention of foundational knowledge, while a higher learning rate was used for the newly initialised conditioning layers to accelerate adaptation to the new task.

Sequence-Level Conditioning

We first implemented two methods that modify the model’s input sequence, serving as comparative baselines to the tailored attention methods.

The Raw Conditioning method involves the direct, digit-level tokenisation of the numerical values in the property vector c∈RP\mathbf{c} \in\mathbb{R}^{P}. This resulting sequence of MM tokens (where M≥PM \geq P, one token per digit for each condition) is prepended to the tokenised input sequence x=(x1,…,xL)\mathbf{x} = (x_1,\ldots,x_L). Formally, if we denote the tokenised property string as tprops=(t1,…,tM)\mathbf{t}_{\mathrm{props}} = (t_1,\ldots,t_M), the new input sequence becomes x′=(⟨bos⟩,t1,…,tM,\n,x1,…,xL,⟨eos⟩)\mathbf{x}' = (\langle\mathrm{bos}\rangle,t_1,\ldots,t_M,\backslash\mathrm{n},x_1,\ldots,x_L,\langle\mathrm{eos}\rangle). As this approach only manipulates the input sequence, it requires no architectural modifications and handles the property values as discrete tokens.

From the hypothesis that a continuous embedding for the properties is more naturally suited to the task, the Prepend Conditioning method introduces a learned, continuous representation of c\mathbf{c}. A Prepend Encoder, implemented as a multi-layer perceptron (MLP) with ReLU activation, projects c\mathbf{c} into a sequence of PP condition embeddings, Ec∈RP×demb\mathbf{E}_c \in\mathbb{R}^{P \times d_{\mathrm{emb}}}, where dembd_{\mathrm{emb}} is the model’s embedding dimension. The input to the transformer is the concatenation of these condition embeddings with the standard token embeddings, Ex∈R(L+P)×demb\mathbf{E}_x \in\mathbb{R}^{(L+P)\times d_{\mathrm{emb}}}. The regular attention mechanism (AoutA_{\mathrm{out}}) operates on this concatenated sequence:

Aout(Qseq,Kseq,Vseq)=softmax⁡(QseqKseqTdk)Vseq,(1)A_{\mathrm{out}}(Q_{\mathrm{seq}},K_{\mathrm{seq}},V_{\mathrm{seq}})=\operatorname{softmax}\left(\frac{Q_{\mathrm{seq}}K_{\mathrm{seq}}^{\mathsf{T}}}{\sqrt{d_k}}\right)V_{\mathrm{seq}}, \tag*{(1)}

where QseqQ_{\mathrm{seq}}, KseqK_{\mathrm{seq}} and VseqV_{\mathrm{seq}} are the Query, Key and Value matrices, respectively, generated from the transformer’s representation of the concatenated sequence containing both token and condition embeddings. The full architecture diagram can be found in Appendix VI H.

Tailored Attention Conditioning

Direct manipulation of the attention mechanism via continuous property representations is hypothesised to enhance targeted generation and mitigate transfer learning disruption more effectively than sequence-level interventions [41]. To evaluate this, the Prefix and Residual architectures were developed to embed property information directly into the transformer attention layers. Intuitively, the Prefix architecture extends the effective context window by concatenating learned property keys and values to the sequence, imposing a “hard” structural bias. On the other hand, the Residual architecture operates as an additive modification to the base self-attention, injecting a parallel property-based score dynamically. This is effectively a weighted residual term that “softly” steers the generation without disrupting the intrinsic sequence attention operations.

Prefix attention conditioning (Figure 2) directly injects property information into the self-attention mechanism of each transformer layer, bypassing sequence encoding. “Ghost” Key-Value (KV) pairs generated from the condition vector are concatenated with input KV pairs. This confines sequence expansion entirely to attention layers rather than FFNNs and input tokens to minimise disruption. The method is a materials-oriented adaptation of Li and Liang’s Prefix Tuning natural language method [41].

Schematic of the Prefix and Residual attention conditioning mechanisms

Figure 2. Schematic of the Prefix and Residual attention conditioning mechanisms. The left panel shows the GPT-2 backbone, where a conditioning encoder (Appendix VI G) injects learned property key-value representations into each of the NN decoder blocks. In Prefix attention (bottom right), QseqQ_{\mathrm{seq}} attends over concatenated property and sequence keys and values, [KPrefix;Kseq][K_{\mathrm{Prefix};}K_{\mathrm{seq}}] and [VPrefix;Vseq][V_{\mathrm{Prefix};V_{\mathrm{seq}}}]. In Residual attention (top right), the base self-attention output AbaseA_{\mathrm{base}} is combined with a parallel property-conditioned term AResidualA_{\mathrm{Residual}} through the learned scaling parameter α\alpha. NN denotes the number of decoder blocks, hh denotes the number of attention heads and α\alpha corresponds to Equation (3). These mechanisms condition generation by either extending or augmenting the attention pathway.

The Prefix Encoder (See Appendix VI G) transforms the input condition vector c∈RP\mathbf{c} \in\mathbb{R}^{P} into a tensor compatible with the attention mechanism. The vector c\mathbf{c} is first projected to a higher-dimensional latent space via a linear layer, followed by Layer Normalisation and ReLU. This hidden representation is projected to the KV tensor dimension for all attention heads and layers, hKV∈RP×N×h×2×dk\mathbf{h}_{\mathrm{KV}} \in\mathbb{R}^{P\times N\times h\times2\times d_k}, where PP is the number of conditions, NN being the number of decoder blocks, hh the number of attention heads and 2×dk2 \times d_k the dimension of the KV pair matrix. The resulting tensor is split into Key (KPrefixK_{\mathrm{Prefix}}) and Value (VPrefixV_{\mathrm{Prefix}}) tensors.

For each layer, the attention mechanism is modified. Given QseqQ_{\mathrm{seq}} from the input sequence, the attention output is computed as:

Aout(Qseq,K,V)=softmax⁡(Qseq(KT)dk)V,(2)A_{\mathrm{out}}(Q_{\mathrm{seq}},K,V)=\operatorname{softmax}\left(\frac{Q_{\mathrm{seq}}(K^{\mathsf{T}})}{\sqrt{d_k}}\right)V, \tag*{(2)}

where K=[KPrefix;Kseq]K=[K_{\mathrm{Prefix}};K_{\mathrm{seq}}] and V=[VPrefix;Vseq]V=[V_{\mathrm{Prefix}};V_{\mathrm{seq}}] are the concatenations of the conditioning- and sequence-derived Key and Value matrices, respectively. The dot product is performed along the sequence length dimension with identical inner embedding dimensions.

Residual attention conditioning (Figure 2) introduces a more nuanced, “soft” conditioning mechanism that dynamically weights the influence of the property conditions. The Residual Encoder (See Appendix VI G) is architecturally similar to the Prefix Encoder but employs tanh activations and utilises reshaping and einsum operations for more structured transformations. These are engineered so that each of the PP conditions initially influences a dedicated portion of the latent space before being combined. The output of the encoder is a set of KV pairs, KResidualK_{\mathrm{Residual}} and VResidualV_{\mathrm{Residual}}, where each pair can be attributed to an input condition. This structured encoding preserves the associativity between each of the PP conditions and its resulting KV pair, distinct from the Prefix approach, where property representations are collectively projected.

The core of this method is the Residual Attention module. For each attention head, two separate attention scores are computed in parallel: a standard self-attention score over the input sequence (AbaseA_{\mathrm{base}}) and a “Residual” attention score (AResidualA_{\mathrm{Residual}}) between the input sequence’s Queries (QseqQ_{\mathrm{seq}}) and the Residual Keys (KResidualK_{\mathrm{Residual}}) and Values (VResidualV_{\mathrm{Residual}}). These are then combined via a learned, per-layer weighting parameter, α\alpha. The modified attention output, AoutA_{\mathrm{out}}, is computed as:

Abase=softmax⁡(QseqKseqTdk)Vseq,AResidual=softmax⁡(QseqKResidualTdk)VResidual,Aout=Abase+α⋅AResidual.(3)\begin{aligned} A_{\mathrm{base}} &= \operatorname{softmax}\left(\frac{Q_{\mathrm{seq}}K_{\mathrm{seq}}^{\mathsf{T}}}{\sqrt{d_k}}\right)V_{\mathrm{seq}},\\ A_{\mathrm{Residual}} &= \operatorname{softmax}\left(\frac{Q_{\mathrm{seq}}K_{\mathrm{Residual}}^{\mathsf{T}}}{\sqrt{d_k}}\right)V_{\mathrm{Residual}},\\ A_{\mathrm{out}} &= A_{\mathrm{base}}+\alpha\cdot A_{\mathrm{Residual}}. \tag*{(3)} \end{aligned}

The scalar weight, α\alpha, is initialised at zero, similar to the “zero-initialisation” technique used in Low-Rank Adaptation (LoRA) fine-tuning [42]. This ensures that the model initially relies only on the pre-trained self-attention mechanism, preserving its generative capabilities and mitigating catastrophic forgetting. Optimisation of α\alpha during training dynamically calibrates the degree of conditioning.

This architecture handles missing or unspecified conditions, a capability enabled by the Residual Encoder’s associativity and the parallel, additive structure of the attention layer. When a value of −100-100 is used in the condition vector c\mathbf{c}, the attention scores (computed as Q⋅KResidualTQ\cdot K_{\mathrm{Residual}}^{\mathsf{T}}) for queries attending to that specific condition variable are set to −∞-\infty, effectively nullifying its contribution after softmax in AResidualA_{\mathrm{Residual}}, similarly to causal attention masking. The additive design bypasses an important limitation of Prefix conditioning for masking “missing” conditions. By concatenating keys, Prefix still changes the sequence length in the attention layers, altering the softmax normalisation. The residual architecture avoids this perturbation, allowing “missing” condition handling while maintaining the crystal sequence’s intrinsic attention distribution.

Validation of the Conditioning Frameworks

Having established the four conditioning architectures, we now evaluate their ability to transfer knowledge from unsupervised learning on unconditional CIF generation to the conditional property-labelled setting. The studies quantitatively compare the efficacy of sequence-level versus attention-based conditioning and how the most promising methods respond to different dataset sizes, representing realistic research settings.

Advantages of pre-training and condition encoding

Figure 3 shows how our four conditioning architectures handle specific band gap targets. We tested the models across a range from 0 eV to 12 eV, which pushes past the high-density region (0 eV to 6 eV) of the training set. We evaluated performance using structural Validity and the Valid-SUN quality metric (QVSUNQ_{\mathrm{VSUN}}) under 2 training regimes. First, models trained from scratch on the MP Bandgap dataset (53.3K structures, with 95% containing 1 to 104 atoms, see Appendix VI B) versus models pre-trained on the broader LeMaterial dataset (4.35M structures, no band gap label) with fine-tuning on the MP Bandgap dataset. Appendix Table VII summarises the quantitative impact of this pre-training.

Output-space distributions for target band gap conditioning with and without pre-training

Figure 3. Output-space distributions for target band gap conditioning with and without pre-training. a. Models trained from scratch. b. Pre-trained models. For each requested band gap, coloured bars distinguish the Prefix, Residual, Prepend and Raw architectures. Lighter bars denote structurally valid generations and darker overlaid bars denote the subset satisfying the comprehensive QVSUNQ_{\mathrm{VSUN}} metric (Validity, Stability, Uniqueness and Structural Novelty; Methods V E). Dashed lines mark the requested target band gaps, and the grey background shows the training-set band gap density. The 12.1 eV target is omitted because no qualifying generations were obtained at this value. Pre-training improves the QVSUNQ_{\mathrm{VSUN}} output space, but performance remains constrained by the labelled-data support.

Unsupervised pre-training improves generative capacity for generating Valid structures across all continuous architectures in this benchmark. On average, aggregate validity increases by 20.4% and the number of QVSUNQ_{\mathrm{VSUN}} structures increases by 14.3%. In particular, the pre-trained models vastly outperform scratch models in terms of Valid structure generation in the tails of the target distribution. The accompanying rise in QVSUNQ_{\mathrm{VSUN}} MAE is consistent with a stronger structural prior that favours plausible configurations, particularly in data-sparse regions.

Target-space analysis (Appendix Figure 14) assesses the correlation of the property of interest to the conditioning value and the Valid structure generation rate to the underlying density of materials. While property hit rates decrease as targets move into sparse data regions (r=0.78r = 0.78), QVSUNQ_{\mathrm{VSUN}} is more stable (r=0.22r = 0.22). Pre-trained models still generate QVSUNQ_{\mathrm{VSUN}} materials near the 6.5 eV target despite low training density there. This indicates that the model can still use sparse boundary regions to find new Valid and QVSUNQ_{\mathrm{VSUN}} materials near the upper tail of the labelled distribution.

At an extreme stress test of 12.1 eV, no architecture produced valid structures. This region is outside of the support of the model, meaning that there are effectively no property-structure mappings to train on. Materials with very large band gaps (> 12.1 eV) are exceedingly rare and have to be highly ionic, with the widest reported experimental band gap being LiF at 12.1 eV [43]. Additionally, property labels are calculated with GGA-DFT which systematically underestimates band gaps by 30–40% [44, 45]. Together, these make the lack of valid generations at this target unsurprising.

To benchmark our approach outside a transformer-only setting, we compared against MatterGen [20], a mixed discrete-continuous diffusion model which operates on graph data, which we will refer to as a graph diffusion model in this work. As shown in Table VII, there is a trade-off between volume and precision. MatterGen generates a greater total number of Valid structures (NValid=6585N_{\mathrm{Valid}} = 6585), but the conditional autoregressive models show better calibration to requested properties, achieving higher QVSUNQ_{\mathrm{VSUN}} R2R^2 values and lower QVSUNQ_{\mathrm{VSUN}} MAEs. The failure at 12.1 eV is also present with MatterGen [20], as both plateau near the upper edge of the labelled support. Finally, the autoregressive pipeline is substantially more efficient in this study’s setting: requiring less GPU time, inference time and peak video random access memory (VRAM) than MatterGen, highlighting that text-based models are more efficient when modelling larger, more complex systems. For a detailed comparison between both conditional generation frameworks, we refer the reader to the discussion in Appendix VI § 1.5.12.

RegimeModelNValidN_{\mathrm{Valid}}R2R^2MAE (±\pm SE)
MatterGen65850.561.58 (0.039)
ScratchPrefix37880.601.03 (0.093)
ScratchResidual40360.611.40 (0.099)
ScratchPrepend41620.651.30 (0.098)
ScratchRaw9560.221.27 (0.383)
Pre-tndPrefix57590.591.48 (0.047)
Pre-tndResidual54570.511.69 (0.056)
Pre-tndPrepend56700.581.46 (0.046)

Table VII. Band gap conditioning performance across Scratch and Pre-trained (Pre-tnd) learning regimes. The autoregressive architectures are compared with the graph diffusion baseline MatterGen [20]. NValidN_{\mathrm{Valid}} is the total number of structurally valid generations. R2R^2 and MAE ±\pm SE are computed between requested and predicted band gaps for QV ⁣SUNQ_{\mathrm{V\!SUN}} materials only. Bold and italics denote the best and second-best autoregressive results within each regime.

The Raw baseline fails during transfer learning and is excluded from further analysis. Concatenating tokens works poorly in low-data regimes, leading to a complete loss of structural validity after pre-training. We focus the rest of our study on the three continuous encoding architectures: Prepend, Prefix and Residual.

Advantages of attention-level conditioning

This section investigates continuous conditioning mechanisms via the physical density property, using the MatterGen Density fine-tuning dataset (653K labelled structures; see Appendix VI A). Density provides an effective benchmark because it is available a priori for all materials. Scaling regimes for generative performance are quantified in Figure 4 and Appendix VI § 1.5.11. The analysis isolates an important transition: at extreme data scarcity, outputs reflect the broad statistical mean of the unconditional prior, but increasing the data volume and specifically the inclusion of long-tail distributions, allows for precise control over target properties.

Image not extracted

Figure 4. Density calibration across fine-tuning set sizes. Rows correspond to the Prefix, Residual and Prepend conditioning architectures, and columns correspond to 1K, 10K, 100K and Full fine-tuning sets. Box plots summarise the generated density distributions for QVSUNQ_{\mathrm{VSUN}} structures at each requested target density. The dashed diagonal marks perfect calibration, and the grey-shaded distribution shows the density distribution of the corresponding training subset on a log scale. Calibration is weak in the 1K and 10K regimes and becomes well aligned with the target values at 100K and Full data volumes.

Within the restricted 1K and 10K regimes, all architectures struggle to deviate from the pre-training distribution. Because these subsets are derived via random sampling, they inherently capture the dense centre of the structural manifold, limiting the variance necessary to learn the extreme examples. In the 1K regime, the Residual method exhibits high structural robustness, generating the maximum number of Valid structures (NValid=7170N_{\mathrm{Valid}} = 7170). The architecture preserves foundational structural knowledge during severe data scarcity. This preservation prevents the immediate collapse of the generative distribution, an effect supported by the loss landscape analysis detailed in Appendix VI § 1.5.13. The Prepend method exhibits the lowest validity and targeting yields in these environments, suggesting that non-attention-level conditioning requires more data to overcome the sequence-level perturbation. Under these regimes, the Prefix model outperforms the other two methods significantly, but remains severely restricted by the lack of structure-property materials present in the tails of the complete dataset.

As the dataset scales to 100K and Full volumes, the architectures successfully overcome the unconditional bias. The Prefix mechanism exhibits the strongest calibration metrics, with the highest positive correlation (QVSUNQ_{\mathrm{VSUN}} R2=0.83R^2 = 0.83 and 0.97) while maintaining the lowest QVSUNQ_{\mathrm{VSUN}} MAE (1.98 g/cm3^3 and 0.72 g/cm3^3) at 100K and Full sizes respectively. On the other hand, the Prepend method required the most data to overcome sequence disruption but ultimately anneals to highly competitive performance, marginally exceeding Prefix and Residual in Q-Hit yield (16.51%) at the Full data limit.

While specific architectural advantages emerge at the extremes of data availability, the aggregate results suggest that Prefix is the strongest overall architecture in this study. It attains the best value in 15 of the 20 primary performance metrics considered in this scaling study, compared with 3 for Residual and 2 for Prepend. Taken together, these results suggest a regime-dependent pattern: Residual conditioning knowledge transfer from pre-training is more robust in the data-limited setting, whereas Prefix provides the best overall trade-off between validity, calibration and data efficiency across the broader study.

Exploration of photovoltaic efficiency chemical space

The Spectroscopic Limited Maximum Efficiency (SLME) serves as a theoretical measure of photovoltaic potential, integrating absorption profiles, band gap characteristics and nonradiative recombination losses to estimate the upper limit of power conversion efficiency [46, 47]. Identifying materials with high SLME requires navigating a multi-dimensional optimisation landscape where both electronic structure and optical properties must be simultaneously considered. This section demonstrates how the CrystaLLM-π\pi framework leverages large-scale unsupervised pre-training to map complex structure-property relationships within a data-scarce regime. The model was fine-tuned on the MP SLME dataset [48], with 5.35K SLME labelled inorganic structures. To steer the generation toward high-performance regions of chemical space, the model was conditioned on the maximum observed SLME in the training set (33.2%). Because the target property set consists of a single, scalar condition with no missing values and fine-tuning data was scarce, prefix conditioning was employed.

Table I details the generation of VSUN structures. Analysis of the compositional novelty profiles reveals distinct behaviour depending on the reference dataset. While 15,462 generated structures possess reduced formulas absent from the fine-tuning corpus, this yield decreases to 1,809 when compared against the expansive pre-training corpus. This disparity is consistent with the transfer of structure-property information from the fine-tuning task onto a broader structural prior acquired during pre-training.

VSUN specs.NN Materials
Fine-tuning, Structural16,463
Fine-tuning, Compositional15,462
Pre-training, Structural8,712
Pre-training, Compositional1,809

Table I. Counts of Valid, Stable, Unique and Novel (VSUN) materials under different novelty definitions and reference datasets (Methods V E). Counts are computed from 100K generations. The fine-tuning dataset is MP SLME with 5.3K structures and the pre-training dataset is LeMaterial with 4.35M structures. Structural novelty denotes the absence of a matching structure under pymatgen’s StructureMatcher, whereas compositional novelty denotes the absence of a material with the same reduced formula and structure in the reference dataset.

To validate the physical properties of the conditionally generated chemical space, the predicted band gaps of the candidate structures are analysed (Figure 5a). Maximising single-junction photovoltaic efficiency requires a band gap between 1.2 eV and 1.4 eV [46]. The baseline training data exhibits an average band gap between 3 eV and 4 eV. If the framework failed to capture the structural determinants of SLME, generation outputs would default to this underlying prior. Instead, the distribution of generated structures shifts away from the dataset mode and clusters within the optimal energy range.

Photovoltaic candidate screening and representative validated structures

Figure 5. Photovoltaic candidate screening and representative validated structures. a. Predicted surrogate Spectroscopic Limited Maximum Efficiency (SLME) is plotted against predicted HSE06 band gap for generated structures, with marginal distributions comparing the generated and training datasets. Dashed reference lines mark the ideal band gap region near 1.2 eV to 1.4 eV and the target SLME of 33.2%. The generated distribution concentrates near the optimal band gap range while spanning a broad range of predicted efficiencies. b. Four Density Functional Theory (DFT) validated candidate structures are shown: Cs₂NaInAs₂ (SLME 26.4%, EhullPBE=−0.605E_{\mathrm{hull}}^{\mathrm{PBE}} = -0.605 eV/atom), Cs₂NaGaAs₂ (SLME 24.4%, EhullPBE=−0.589E_{\mathrm{hull}}^{\mathrm{PBE}} = -0.589 eV/atom), NaHfCuS₃ (SLME 23.3%, EhullPBE=−0.013E_{\mathrm{hull}}^{\mathrm{PBE}} = -0.013 eV/atom) and Rb₂(NbBr₃)₃ (SLME 13.3%, EhullPBE=0.001E_{\mathrm{hull}}^{\mathrm{PBE}} = 0.001 eV/atom). Rb₂(NbCl₃)₃ structures (SLME: 11.6% and 11.3%, both EhullPBEE_{\mathrm{hull}}^{\mathrm{PBE}}: 0.003 eV/atom, 2 polymorphs not shown) were also identified as stable, promising candidates. The model effectively steered generation to a promising area of chemical space to generate materials of interest.

work failed to capture the structural determinants of SLME, generation outputs would default to this underlying prior. Instead, the distribution of generated structures shifts away from the dataset mode and clusters within the optimal energy range. This shift away from the dataset mode is consistent with the model learning structure-property correlations relevant to high SLME, rather than simply reproducing the dominant band gap distribution of the training set.

Following the criteria established by Yu and Zunger [47], an SLME exceeding 20% indicates a high-performance photovoltaic material. Within the 16,463 structurally novel materials with respect to the fine-tuning set (Table I), the generated set counts 1,546 candidates with ALIGNN-predicted SL MEs surpassing the threshold. To manage the computational cost of high-fidelity DFT validation, subsequent analysis focused on a subset of 30 top-ranking candidates. This selection prioritised candidates demonstrating both high predicted SLME and favourable sustainability profiles, quantified via the Herfindahl-Hirschman Index (HHI) for elemental supply chains (see Methods V E). The properties and selection rationale for these candidates are detailed in Table XVI, with four representative structures highlighted in Figure 5b.

Reduced FormulaPred. SLME (%)Sustainability (HHIdistHHI_{dist})Novelty (Str.Comp.)MACE EhullE_{\mathrm{hull}}DFT EhullE_{\mathrm{hull}}DFT SLME (%)Outcome / Status
Cs3(GeBr3)4Cs_{3}(GeBr_{3})_{4}30.87224ft-pt | ft-pt0.040––Excluded: Unusual oxidation states
Rb2(NbCl3)3Rb_{2}(NbCl_{3})_{3}30.47203ft | ft0.0210.00311.6Validated: Stable
Rb2(NbCl3)3Rb_{2}(NbCl_{3})_{3}29.97203ft | ft0.0150.00311.3Validated: Stable
Rb2(NbBr3)3Rb_{2}(NbBr_{3})_{3}29.88747ft-pt | ft-pt0.0170.00113.25Validated: Stable
Cs2(NbCl3)3Cs_{2}(NbCl_{3})_{3}28.97344ft | ft-0.011––Excluded: Structural Redundancy
Ba3GaAs3Ba_{3}GaAs_{3}28.54361ft-pt | ft0.086––Excluded: Structural Redundancy
CsSnTe2CsSnTe_{2}28.45724ft-pt | ft0.072-0.329MetallicInterested: but DFT showed metallic
TaTeI3TaTeI_{3}28.46166ft | ft0.024––Excluded: Unusual oxidation states
Cs2NaInAs2Cs_{2}NaInAs_{2}28.16298ft | ft-0.070-0.60526.4Validated: High Efficiency and Stable
TaI6TaI_{6}28.16512ft-pt | ft0.016––Excluded: Unusual oxidation states
Ba4Bi2OBa_{4}Bi_{2}O27.95502ft | ft0.015––Excluded: Low Novelty
K3InAs2K_{3}InAs_{2}27.95204ft | ft-0.013––Excluded: Structural Redundancy
Cs2NaGaAs2Cs_{2}NaGaAs_{2}27.96731ft | ft-0.042-0.58924.4Validated: High Efficiency and Stable
K2NaSnSb2K_{2}NaSnSb_{2}27.96198ft | ft0.0000.035MetallicInterested: but DFT showed metallic
Cs2NaSnAs2Cs_{2}NaSnAs_{2}27.86114ft | ft0.008––Excluded: Structural Redundancy
Condition: Combined Sustainability (HHIdistHHI_{dist}) and SLME Optimisation
KAg3S2KAg_{3}S_{2}25.52207ft | None0.045––Excluded: Low Novelty
Cu3GeS4Cu_{3}GeS_{4}25.82440ft | ft-0.032––Excluded: Low Novelty
KCu3S2KCu_{3}S_{2}25.22574ft-pt | ft0.029––Excluded: Chemical Heuristics (Cu-II)
SnSSnS25.52644ft-pt | None0.057––Excluded: Known Material
Ba(CuS)2Ba(CuS)_{2}25.82656ft | None0.057––Excluded: Low Novelty
K2Ni3S4K_{2}Ni_{3}S_{4}25.42706ft | None0.001––Excluded: Low Novelty
K2(TiCl3)3K_{2}(TiCl_{3})_{3}27.12748ft | ft0.026––Excluded: Unusual oxidation states
K2(TiCl3)3K_{2}(TiCl_{3})_{3}26.02748ft | ft0.018––Excluded: Unusual oxidation states
K2(TiCl3)3K_{2}(TiCl_{3})_{3}26.52748ft | ft0.022––Excluded: Unusual oxidation states
Cu2SiSe4Cu_{2}SiSe_{4}26.02794ft-pt | ft0.047––Excluded: Chemical Heuristics (Cu-II)
K2Cu3S4K_{2}Cu_{3}S_{4}26.42797ft-pt | ft-pt0.0640.183MetallicInterested: but DFT showed metallic
NaHfCuS3NaHfCuS_{3}25.12886ft | ft-0.076-0.01323.3Validated: High Efficiency and Stable
Ba(CuSe)2Ba(CuSe)_{2}26.22974ft | None0.045––Excluded: Low Novelty
KCu2Se3KCu_{2}Se_{3}26.93014ft-pt | ft0.052––Excluded: Chemical Heuristics (Cu-II)
KCu2Se3KCu_{2}Se_{3}26.33014ft | ft0.048––Excluded: Chemical Heuristics (Cu-II)

Table XVI. Screening summary of the top 30 generated photovoltaic candidates sorted by their conditioning metric. Bold entries denote candidates validated by DFT as stable or high-efficiency, while entries with outcome status in italics denote materials that were investigated further with DFT but did not remain satisfactory. The top panel lists candidates conditioned purely on optimal SLME (> 30%), while the bottom panel targets a multi-objective condition of high SLME and low supply-chain risk (Methods V.E). Novelty is reported relative to the fine-tuning dataset only (ft) or to both the fine-tuning and pre-training datasets (ft-pt). “Unusual oxidation states” exclusions indicate cases where charge balance would require chemically unusual oxidation states. Cu(II)-containing compounds are excluded because3 copper is difficult to treat accurately with DFT. EhullE_{\mathrm{hull}} values are reported in eV/atom. Following Yu and Zunger, materials with SLME above 20% are classified as high efficiency [47].

Analysis of the generated candidates reveals that the model output distribution exhibits chemically intuitive optimisation pathways, such as atomic substitutions on established high-performance motifs. For instance, the generation of Cs2NaInAs2\mathrm{Cs}_{2}\mathrm{NaInAs}_{2} and Cs2NaGaAs2\mathrm{Cs}_{2}\mathrm{NaGaAs}_{2} exploits the optoelectronic properties of known InAs and GaAs motifs [49, 50] to propose stable, high-SLME quaternary variants. Furthermore, the model identified a metastable family of rubidium-niobium halides, including Rb2(NbBr3)3\mathrm{Rb}_{2}(\mathrm{NbBr}_{3})_{3} (EhullPBEE_{\mathrm{hull}}^{\mathrm{PBE}}: 0.001 eV/atom) and two polymorphs of Rb2(NbCl3)3\mathrm{Rb}_{2}(\mathrm{NbCl}_{3})_{3} (EhullPBEE_{\mathrm{hull}}^{\mathrm{PBE}}: 0.003 eV/atom). These compositions are completely novel relative to the fine-tuning dataset, and the bromide variant is entirely absent from all training sets. While ALIGNN surrogate predictions initially suggested promising SLME values for these halides, subsequent hybrid DFT characterisation revealed that the conduction band minimum is split off from the main conduction band density of states. This electronic structure results in poor absorption properties and potentially low electron conductivity. This discrepancy highlights an important limitation of surrogate property-prediction models for materials discovery and the importance of ab initio validation. Finally, the framework proposed NaHfCuS3\mathrm{NaHfCuS}_{3}, a chalcogenide variant with a validated SLME of 22.6%. Although chalcogenides are established in thin-film photovoltaics [51, 52], this specific material lacks prior photovoltaic investigation to the best of our knowledge. While high-fidelity validation was restricted to a curated subset, the broader pool of over 1,546 structurally novel candidates with predicted high SLME provides a substantial chemical space for future high-throughput characterisation.

Recovery of materials using X-Ray Diffraction (XRD) information conditioning

Benchmarking against other ML models for XRD conditioned generation

To validate CrystaLLM-π\pi on recovery tasks, its performance in XRD-conditioned structure determination was benchmarked against concurrently developed generative architectures. These baselines included DiffractGPT, a general-purpose large language model fine-tuned for XRD data via prompt engineering [31]; Uni3Dar, an autoregressive model which uses compressed spatial representations with multimodal conditioning tokens [53]; and PXRDGen, a diffusion pipeline employing contrastive learning to guide generation [54]. Comparative evaluations were conducted using benchmarks on established MP-20 [55] and Jarvis-DFT [56] datasets.

For the MP-20 benchmark, CrystaLLM-π\pi was trained from scratch following the PXRDGen protocol. With a perplexity-based ranking for candidate selection, the model achieves a conditional match rate of 62.73% with an RMSD of 0.0444 Å (Table II). As noted in the PXRDGen paper [54], some of the test set structures fail standard validity checks applied post-generation, meaning that they could never be matched under these checks. Therefore, we also performed an analysis with a relaxed validity check for proposed structures (“Skip Valid.”), which shows an elevated match rate of 69.01%.

ModelConditional Match %Conditional RMSDUnconditional Match %Unconditional RMSD
PXRDGen68.680.0707
Uni3Dar75.080.027665.480.0317
CrystaLLM-π\pi62.730.044455.850.0437
CrystaLLM-π\pi (Skip Valid.)69.010.0465--

Table II. Performance on the MP-20 benchmark for XRD-guided crystal structure prediction. Bold and italics denote the best and second-best metrics. Each ground-truth structure is evaluated against a single reported candidate per method. PXRDGen uses a 1-shot generation protocol, whereas Uni3Dar and CrystaLLM-π\pi use “1-perplexity” ranking. Unconditional CrystaLLM-π\pi results are taken from the reported “1-shot” MP-20 benchmark in Antunes et al. [38]. The “Skip Valid.” row reports results when the benchmark validity screen is bypassed during matching.

The framework was evaluated on the Jarvis-DFT benchmark following the DiffractGPT protocol [31]. To establish reproducible evaluation conditions, CrystaLLM-π\pi was fine-tuned using a seeded randomised 90:10 train-test data split. Maintaining parity with the baseline Jarvis-XRD benchmark, pre-matching validity screens were excluded during evaluation to prevent arbitrary filtering of potential matches.

Table III compares performance against Gradient Boosting (GBR), CNN and DiffractGPT baselines. CrystaLLM-π\pi demonstrates improved structural accuracy: the top perplexity-ranked structure (Section V D) achieves an RMSD of 0.0347 Å, representing a 50.4% error reduction relative to DiffractGPT (0.0700 Å). This fidelity is confirmed by the lowest Mean Absolute Error (MAE) across all lattice parameters. When evaluating the best match drawn from a pool of “20-consistent” structures, where a structure is deemed consistent if the output CIF is syntactically sensible (See Methods V D), the match rate reaches 85.47% while maintaining a 0.0361 Å RMSD.

A stratified analysis assessed generalisation by classifying the test set into materials observed during training phases (“Seen”), structurally novel polymorphs of known compositions (“Novel Polymorphs”), and materials with entirely unseen chemical compositions (“Novel Compositions”; Appendix VIO). Performance was broadly stable across these subsets, with novel compositions showing slightly better aggregate fidelity and match rates in this dataset. Appendix VIO discusses likely contributors to this pattern, including sample-composition effects.

Experimental Structure Recovery from Chili-100K Data

The model was evaluated on a 1.5K-material test set drawn from the Chili-100K XRD dataset. The underlying crystal structures are experimentally determined COD entries, while the conditioning XRD vectors were simulated from those structures using the same preprocessing pipeline as the other XRD benchmarks. This curated subset spans a broad distribution of atom counts and reflects realistic experimental structure complexity (See Appendix VI B). To assess generative generalisation and control for data leakage, the test set was stratified into three 500-material subsets, as discussed in the previous section: “Seen”, “Novel Polymorph”, and “Novel Composition”. The XRD-conditioned model was compared against a mirrored unconditioned baseline model trained with identical datasets and hyperparameters, but without XRD conditioning entirely.

ModelMatch %RMSDMAE aMAE bMAE c
GBR--1.0300.9901.270
CNN--0.2800.2700.280
DiffractGPT (formula)-0.07000.1700.1800.270
CrystaLLM-π\pi (20-consistent)85.470.03610.1420.1350.216
CrystaLLM-π\pi (1-perplexity)66.220.03470.0960.0980.183

Table III. Performance on the Jarvis-DFT benchmark. Bold and italics denote the best and second-best metrics, respectively. CrystaLLM-π\pi results are reported for the highest-ranked perplexity output (“1-perplexity”) and for the best structural match within 20 attempts (“20-consistent”). MAE a, MAE b and MAE c are absolute errors in the lattice parameters of the matched structures.

Results from three independent test-set recovery experiments (Figure 6) support the hypothesis that diffraction pattern conditioning provides improved structure generation. When conditioned on XRD profiles and evaluated over “20-consistent” generations, the model achieved a mean structure match rate of 49.04%, compared to 15.89% for the unconditioned model. Within this benchmark, the stratified results are consistent with strong generalisation across seen, structurally novel and compositionally novel subsets. The structurally novel subset had the highest recovery rate; this improvement over fully seen materials is likely explained by the lower geometric complexity of the unseen polymorphs. Furthermore, for entirely novel compositions, the conditioned model maintained a 41.40% match rate, while the unconditioned baseline degraded to 8.87%. Figure 6 shows a tighter correlation between predicted and target values across all test structures for the conditioned models. The unconditional model deteriorated on larger systems, exhibiting a strict performance ceiling that prevented the recovery of any structures exceeding 40 atoms per unit cell. In contrast, the conditioned model successfully recovered structures up to 83 ±\pm 10 (SE) atoms, raising the average matched structure from 11.03 atoms in the unconditioned setting to 19.13 atoms per unit cell.

Crystal structure recovery plots

Figure 6. Crystal structure recovery on the Chili-100K benchmark with and without XRD conditioning. Top: Predicted versus experimental lattice parameters (aa, bb, cc) and unit-cell volumes are shown for the XRD-conditioned model (left four panels) and the text-only baseline without XRD information (right four panels). Point colour indicates whether the best candidate from “20-consistent” generations matches the target structure, point size scales with the target atom count and dashed lines denote perfect agreement. The text-only baseline does not produce format-compliant CIFs for targets above 40 atoms, which explains their absence from the right-hand panels. Bottom: Recovery metrics are reported for models with and without XRD conditioning under “20-consistent” and “1-perplexity” generation strategies, stratified into seen, structurally novel and compositionally novel test subsets. Reported quantities are match rate, mean RMSD, volume mean absolute error (MAE), average matched atoms, maximum matched atoms, sequence perplexity and total generation time per target under a 200-attempt budget on the 1.5K-material test set. Values are reported as mean ± standard error over three independent runs. XRD conditioning improves recovery accuracy and extends recovery to larger crystal systems.

Because diffraction patterns explicitly encode lattice dimensions and crystallographic symmetry, this conditioning provides global structural priors. These priors guide the autoregressive generation process, mitigating the error accumulation typically observed in long sequence generation. Generating “20-consistent” structures across the test set was 2.75-times quicker in inference time for conditioned, compared to unconditioned models (Figure 6). This efficiency gain is supported by reduced model uncertainty during autoregression: the unconditioned baseline exhibited an average sequence perplexity of 1.312 compared to 1.247 with XRD signal, showing the conditioning’s capability to steer generation by restricting the probability distribution at each generation step.

Experimental Structure Recovery for TiO2_2 polymorphs

Resolving structurally distinct polymorphs sharing identical reduced formulas presents a critical challenge for generative crystallography. To evaluate the capacity of the model to synthesise specific geometric configurations from diffraction data, structural recovery was tested on three experimental TiO2_2 polymorphs: Rutile, Anatase and Brookite. The Chili-100K training dataset contained no occurrences of these phases, with Rutile appearing only in the validation split. The un-labeled pre-training dataset contained only Anatase and Rutile, meaning Brookite was never seen at any training phase. Furthermore, the unit cell stoichiometries for Anatase (Ti4_4O8_8) and Brookite (Ti8_8O16_{16}) are not unique to these specific phases within the broader pre-training distribution. Accurate recovery, therefore, requires the model to actively interpret the XRD conditioning to differentiate the target topology from competing structures with identical atom counts.

The Chili-100K model successfully recovered all three polymorphs using only the input composition and the experimental XRD profile (Table IV). While match rates decline for Anatase and Brookite, producing Valid structural targets within a 20-candidate generation budget represents a tractable screening volume for experimental validation. The recovery of the entirely unseen Brookite phase supports the interpretation that the model can generate structures guided by the diffraction signal rather than relying only on memorised chemical distributions.

To test whether the model learned a direct physical mapping rather than a compositional lookup table, inference was performed targeting 2×\times expanded supercells. The model was prompted with doubled stoichiometry and the corresponding complex XRD pattern. Successful recovery of Rutile and Anatase under doubled stoichiometry suggests that the model is responding to the composition-XRD constraint pair rather than performing a simple compositional lookup. The model was unable to match the Brookite phase, although it generated Valid outputs, likely due to the increased complexity of a lower symmetry and high atom count system.

Discussion and limitations

The central result of this study is that continuous control in autoregressive crystal generation depends strongly on where the conditioning signal enters the network. Generative materials design requires models that can map continuous physical targets onto discrete structural outputs without eroding the structural prior acquired during unsupervised pre-training. In standard autoregressive transformers, this interface is difficult because digit-level property tokenisation weakens ordinal structure and sequence-level conditioning perturbs the token representations that govern CIF generation. CrystalLLM-π\pi addresses this limitation by injecting continuous property information directly into multi-head attention through Prefix and Residual key-value conditioning, rather than encoding targets as additional sequence tokens. Localising adaptation to the attention pathway preserves the original CIF token stream, reduces disruption to the pre-trained backbone, and maintains high structural validity during conditional fine-tuning. The contribution is therefore a conditional crystal-generation framework together with evidence that attention-level conditioning can preserve pre-trained structural priors under scarce labelled supervision. The work also addresses an evaluative gap in the field: conditional generation still lacks a comprehensive benchmark suite spanning transfer learning, data regimes and downstream task types. The controlled comparisons assembled here are intended as a step toward that more systematic standard.

Model SetupData SplitMatch % (SESE)Mean RMSD (SESE)Vol MAE [Å3^3] (SESE)Avg. Atoms (SESE)Max Atoms (SESE)Avg. Perplexity (SESE)Total time (h:m:s)
ConditionedOverall (20-consistent)49.0 (±0.4)0.115 (±0.001)44.9 (±0.6)19.1 (±0.1)83 (±10)1.247 (±0.001)1:07:50 (±0:23)
Seen45.5 (±1.1)0.119 (±0.003)41.4 (±0.6)21.9 (±0.3)83 (±10)1.268 (±0.003)
Structurally Novel60.3 (±0.3)0.120 (±0.002)42.1 (±0.9)15.6 (±0.0)681.205 (±0.002)
Compositionally Novel41.4 (±0.7)0.103 (±0.005)51.1 (±0.7)21.2 (±0.1)65 (±1)1.268 (±0.003)
Overall (1-perplexity)34.4 (±0.2)0.104 (±0.000)53.3 (±0.4)17.4 (±0.2)68 (±2)1.217 (±0.001)
UnconditionedOverall (20-consistent)15.9 (±0.1)0.135 (±0.001)101.7 (±0.2)11.0 (±0.2)401.312 (±0.000)3:07:14 (±0:37)
Seen14.2 (±0.3)0.135 (±0.005)107.1 (±2.4)11.4 (±0.0)401.329 (±0.005)
Structurally Novel24.6 (±0.0)0.160 (±0.003)92.2 (±0.3)10.3 (±0.3)401.264 (±0.001)
Compositionally Novel8.9 (±0.4)0.064 (±0.011)107.2 (±2.8)12.4 (±0.3)401.352 (±0.007)
Overall (1-perplexity)8.3 (±0.3)0.105 (±0.011)165.9 (±2.1)11.7 (±0.3)401.301 (±0.000)

Table 6. Recovery metrics for the Chili-100K benchmark.

A direct comparison with MatterGen (a state-of-the-art diffusion-based generative model) clarifies that the two generative frameworks favour different objectives. In the evaluated band gap setting, the MatterGen baseline produced a larger absolute number of Valid and QVSUNQ_{\mathrm{V S U N}} structures, consistent with the stronger geometric inductive bias of a graph-based diffusion model that operates directly on atomic environments and interatomic relationships. By contrast, CrystaLLM-π\pi maintained tighter calibration to requested targets while requiring less memory and shorter training and inference times. These gains follow from a single-stage autoregressive pipeline in which conditioning is introduced through standard attention operations on a one-dimensional CIF representation, avoiding iterative denoising and graph construction at sampling time. The cost of this efficiency is a weaker built-in geometric prior, because the text model must infer spatial regularities from sequence statistics rather than operating directly on coordinates and neighbourhoods. Even so, the discrete CIF representation preserved crystallographic symmetry more reliably in the unrelaxed outputs examined here and remained practical for larger, chemically richer unit cells. The comparison, therefore, supports complementarity rather than a universal ranking: graph diffusion appears better suited to maximising QVSUNQ_\mathrm{VSUN} structure yield for chemistries of relatively limited complexity, whereas attention-level autoregression offers a lighter route to scalable conditional generation when target precision, symmetry retention and computational cost are the primary constraints.

TiO2_2 PolymorphRutileAnataseBrookite
Atoms (& ×2\times2 SC)6 (12)12 (24)24 (48)
PromptsNN Matches (RMSD)NN Matches (RMSD)NN Matches (RMSD)
Comp.17 (7.70 ×10−4\times10^{-4})2 (3.62 ×10−1\times10^{-1})7 (3.96 ×10−1\times10^{-1})
Supercell Comp.2 (2.00 ×10−1\times10^{-1})4 (3.39 ×10−1\times10^{-1})0

Table IV. Recovery metrics for experimental TiO2_2 polymorphs using the CHILI-fine-tuned model. Rutile, Anatase and Brookite are evaluated under conventional-cell composition prompts and doubled supercell composition prompts (×\times2). Each entry reports the number of matched structures obtained under the “20-consistent” protocol, with the lowest RMSD among the matched structures given in parentheses.

The photovoltaic case study extends the conditioning framework from benchmark calibration to functional discovery in a scarce-label regime. SLME is a demanding target because it compresses absorption strength, band gap and recombination losses into a single scalar objective, so successful steering requires the model to recover indirect structure-property relationships from only 5.35K labelled materials. The results are consistent with a transfer-learning mechanism in which large-scale pre-training supplies a broad structural prior and the conditional objective redirects sampling toward motifs associated with favourable optoelectronic response. The novelty profile supports this interpretation, because many candidates are novel with respect to the fine-tuning set, whereas fewer remain compositionally novel with respect to the pre-training corpus. The DFT follow-up then sets the practical boundary of the claim. Several candidates retained promising photovoltaic characteristics after first-principles validation, whereas others exposed errors in the surrogate ranking when subtle electronic-structure features degraded absorption despite favourable surrogate evaluation. The main implication is that CrystaLLM-π\pi can enrich the search space for plausible high-performance materials under limited labelled supervision, but functional assessment still requires high-fidelity electronic-structure validation.

The XRD experiments show that the same conditioning mechanism can absorb heterogeneous experimental constraints and convert them into useful generative bias. Unlike scalar property targets, diffraction profiles encode distributed information about lattice dimensions, symmetry and periodic ordering, so successful recovery requires the model to align a high-dimensional continuous signal with a discrete CIF sequence generation task. In this setting, conditioning improved structure recovery across theoretical and experimental benchmarks and allowed the model to recover larger and more complex materials compared to an unconditioned baseline.

The generative range of CrystaLLM-π\pi is limited by two factors: the coverage of the training data and the maximum sequence length of the transformer backbone. Attention-level conditioning can steer generation toward rare target regions, but it does not allow the model to move reliably far beyond the chemical space represented in the data it has learned. This limitation is clearest at extreme target values, where physically plausible structures are sparse and extrapolation remains unreliable. In practice, exploring a new region of property space still requires nearby structure-property examples in the training data. When pre-training and fine-tuning do not provide enough local support around a target region, conditional generation in that part of the space becomes less reliable. We demonstrate that this limitation also applies to graph diffusion-based conditional generation. The second constraint arises from the explicit CIF representation. Because sequence length scales with the asymmetric unit rather than the total atom count, large high-symmetry structures can remain tractable, whereas low-symmetry crystals rapidly exhaust the context window. The present framework, therefore, retains a representational bias toward compact crystallographic descriptions and against materials whose disorder or low symmetry requires long token sequences.

These limits define clear next steps. Longer-context backbones or more efficient positional schemes could expand coverage of low-symmetry and compositionally complex structures without losing the practical advantages of autoregressive decoding. Extending the training dataset to contain CIFs with fractional occupancies and substitutional disorder would also bring the model closer to experimental materials, where ideal ordered cells are often an approximation rather than the true state. Another promising direction is to combine autoregressive crystal generation with value-aware numerical representations such as xVal [57], which could improve interpolation and extrapolation for coordinates, lattice parameters and conditioned properties. We do not evaluate such methods here because they would require changing the pre-training objective from pure next-token prediction to a mixed value-and-token learning setup, making them a distinct potential modelling direction. Finally, mechanistic analysis of the conditioned attention layers could reveal how continuous targets alter token probabilities across composition, symmetry and coordinate fields. That level of interpretability would strengthen the scientific value of the framework by turning conditional control from an empirical capability into a tool for studying learned structure-property relations.

Conclusion

We have presented CrystaLLM-π\pi, a conditional autoregressive model for crystal structure generation. We have demonstrated that fine-tuning a pre-trained crystal structure generator yields a model capable of generating new materials with desired characteristics. We have further shown the utility of this capacity in two important materials science settings: the discovery of new materials with target functional properties and the recovery of crystal structures from experimental characterisation data. In both settings, we have shown that CrystaLLM-π\pi offers a lightweight, flexible and extensible framework that achieves state-of-the-art, or near state-of-the-art, performance. We have provided an easy-to-install codebase, together with a pre-built containerised API and a web-app interface, making CrystaLLM-π\pi an accessible and powerful tool for accelerated materials development. Relevant links to these tools are provided in § 1.4 G.

Methods

Datasets

Datasets were curated to study the model under different pre-training and fine-tuning conditions, as shown in Table V. All datasets were stored in dataframes, with each row containing a material’s database of origin, its reduced formula, its augmented CIF string and optional numerical properties of interest along with their normalised counterparts. All the CIF data underwent a standardised preprocessing pipeline including deduplication, data augmentation, tokenisation and specialised batching to incorporate conditioning information, with full details provided in Appendix VI C. For specific use cases, we applied different pre-processing; full details for exact reproducibility can be found in Appendix VI A.

DatasetNb MaterialsConditions
MP Bandgap53.3KBand gap, EhullPBEE_{\mathrm{hull}}^{\mathrm{PBE}}
MatterGen Density653KDensity, EhullPBEE_{\mathrm{hull}}^{\mathrm{PBE}}
MP SLME5.35KSLME
MP-20 XRD45.2KTop-20 XRD
Jarvis-DFT XRD76.0KTop-20 XRD
MatterGen XRD658KTop-20 XRD
Chili-100K XRD14.1KTop-20 XRD

Table V. Summary of property-conditioned datasets used in this work. For X-Ray Diffraction (XRD) conditioning, the condition vector stores the 20 most intense peaks and their associated 2θ2\theta positions. All datasets were filtered to remove CIFs containing more than 1024 tokens, except Chili-100K, which uses a context length of 1536. All processed datasets are openly available at Hugging Face [58].

In the XRD task, methodological standardisation was required to ensure a fair comparison despite the variation within existing benchmarking protocols. As the original data partitions for the Jarvis-DFT benchmark were unavailable, a seeded, randomised 90:10 train-test split was established to facilitate reproducible evaluation. Furthermore, standard validity screens employed in benchmarks such as MP-20 were observed to arbitrarily discard 8.93% of structure-matching test structures, a limitation discussed in the PXRDGen paper [54]. To isolate the intrinsic structural fidelity of the generative mechanisms from these filters, performance metrics on the MP-20 dataset are reported both with and without the filters.

Transformer backbone

In the pre-training phase, the model learns to generate CIFs sequentially. The model backbone is based on CrystaLLM’s small model hyperparameter choices and was pre-trained on an unlabeled augmented CIF dataset [38]. It consists of ∼26\sim26 million parameters, with an embedding size of 512, 8 decoder block layers and 8 attention heads. For functional property conditioning, the pre-trained model was adapted for fine-tuning, where CIFs were labelled with associated properties, such as band gap or density. Architectural modifications, detailed in § 1.1 B, were implemented to guide CIF sequence generation using these properties. The objective of this fine-tuning stage was to enable the model to learn a mapping from a vector of material properties to chemically valid crystal structures. Default model training protocols and hyperparameter choices can be found in Appendix VI § 1.5.16, and the hyperparameter search space is defined for each experiment in Appendix VI § 1.5.17. The final model configuration for every model used in this work can be found in the repository configuration files [59].

The model backbone was trained by minimising a developed variation of the cross-entropy loss, where the commonly used base loss is defined as:

LCE=1N∑n=1N(−∑k=1Kyk,nlog⁡(pk,n)),(4)\mathcal{L}_{CE} = \frac{1}{N}\sum_{n=1}^{N}\left(-\sum_{k=1}^{K}y_{k,n}\log(p_{k,n})\right), \tag*{(4)}

where NN is the total number of tokens in the batch for which the loss is calculated, KK is the size of the token vocabulary, yk,ny_{k,n} is the true probability for token kk at position nn in the sequence, set to 1 for the correct target token and 0 otherwise, pk,np_{k,n} is the model’s predicted probability for token kk at position nn and log⁡\log is the natural logarithm.

The modification incorporates an additional penalty targeting fixed tokens. These tokens are defined as the syntactically invariable elements within the CIF (e.g., “data_”, “_cell_volume”, see Appendix VI D). Conversely, variable tokens are the parts of the CIF that can change from one structure to another (e.g. “Ge4”), which are encapsulated in brackets during CIF augmentation. The penalty on fixed tokens aims to accelerate the learning of CIF syntax during initial training phases, diminishing gradually beyond the initial training phases as the model’s syntactic understanding improves (Appendix VI M). The final loss objective is defined as:

L=n1LCE+n2LFixed,(5)\mathcal{L} = n_{1}\mathcal{L}_{\mathrm{CE}} + n_{2}\mathcal{L}_{\mathrm{Fixed}}, \tag*{(5)}

where LCE\mathcal{L}_{\mathrm{CE}} is the regular cross-entropy loss and LFixed\mathcal{L}_{\mathrm{Fixed}} is the additional cross-entropy loss applied only to Fixed tokens. n1n_{1} and n2n_{2} are tunable hyperparameters to adjust how much each component contributes to the overall loss L\mathcal{L}. For the large-scale pre-training, we set {n1,n2}\{n_{1}, n_{2}\} to {1,2}\{1, 2\}; for the rest of the studies, we set them to {1,1}\{1, 1\}.

Conditioning Method Implementation

To facilitate property-conditioned generation, we set up a dual optimisation strategy during fine-tuning, adapting the AdamW optimiser inspired by Howard and Rude’s recommendations [60, 61]. The parameter set of the model, θ\theta, was split into two subsets: θbackbone\theta_{\mathrm{backbone}}, which contains the parameters of the pre-trained backbone and θcond\theta_{\mathrm{cond}}, which contains the parameters of the newly initialised conditioning layers. This split allows for differential learning rates, with the goal of maximising the information transfer between pre-training and fine-tuning steps while enabling stronger learning of the conditioning task.

The full hyperparameter settings, including hidden dimensions, dropout, the differential learning rates applied to θbackbone\theta_{\mathrm{backbone}} and θcond\theta_{\mathrm{cond}} for fine-tuning, are provided in Appendix VI P. Experiment-specific hyperparameters are specified in the repository configuration files [59].

CIF generation and top perplexity scoring

During model inference, sampling is used to generate tokens sequentially. This work applies sampling temperature T, as well as Top-K and nucleus Top-p sampling schemes to allow dynamic selection of tokens [62, 63]. Tuning them allows sampling from highly likely tokens (reflecting frequent training patterns) to less likely tokens, which introduces exploratory randomness during generation. The exact parameter values differ from experiment to experiment (See Appendix VI I).

For crystal structure generation, text prompts and condition pairs are established, with specified levels of verbosity in the prompts. With the Residual method, for example, we can include any combination of the following: functional property, no cell composition, cell composition, or space group without needing to retrain the model or repurpose the framework. For each prompt-condition pair, the model generates candidate CIF structures using the specified sampling parameters.

A distinct generative workflow is employed for the structure recovery tasks (§ 1.1 E) to maximise prediction fidelity. Outputs undergo three algorithmic consistency checks. Formula consistency verifies a coherent stoichiometry throughout the output CIF. Space-group consistency compares the stated space-group symbol against the symmetry detected by pymatgen with a precision tolerance of 0.1 Å [64]. Sensibility tests reject structures with cell lengths outside 0.5 Å to 1000 Å or cell angles outside 10° to 170°. To prioritise the most physically probable reconstruction, a perplexity-based scoring mechanism evaluates the generated candidates. Perplexity is computed from the token transition scores, excluding the input prompt. Lower values indicate higher model confidence based on learned structural priors. The framework generates candidates until achieving 20 self-consistent CIFs. The “1-perplexity” scheme then selects the single structure with the lowest perplexity for final matching. This approach balances exploration through sampling and composition prompts with condition priors, together with quality control through validation and probabilistic ranking, increasing the model’s capacity to generate sensible structures for a given input (See Appendix VI N for ablations).

In contrast, the discovery of novel materials requires broad exploration of the chemical space. So, perplexity ranking is not used during discovery tasks (§ 1.1 D). Candidates are generated using pure sampling strategies with discovery-oriented parameters (Appendix VI I), and only VSUN physical filters are used to select promising candidates. This ensures the framework applies high-confidence constraints for structure solving while favouring stronger deviation from the training manifold during inverse design.

Evaluation Metrics

To evaluate the developed methods, the following performance metrics are used: CIF validity criteria for chemical consistency and structural reconstruction; structure similarity measures for novelty assessment against training data; uniqueness within generated sets; and statistics for comparative effectiveness of conditioning methods. DFT calculations then provide additional validation of thermodynamic stability and electronic properties for selected candidates.

Validity Criteria

The validity metric follows the criteria of the original CrystalLLM paper [38]. It must be possible for a structure to be reconstructed from the generated CIF, the space group must be consistent with the atomic structure, the generated bond lengths must be reasonable with respect to the atomic radii, and finally, the declared atom site multiplicity needs to be consistent with the unit cell composition.

Structure Diversity

For the uniqueness metric, sets of generated structures with identical input constraints are compared. We use the LeMaterials BAWLHasher (Bonding Algorithm Weisfeiler Lehman), which is a materials fingerprint method based on bonding graph structure, reduced formula and symmetry [36]. Canonical hashes are obtained for each structure in a uniqueness pass, and structures are counted as unique if they have a unique hash. This allows the calculation to scale O(n)O(n), rather than O(n2)O(n^2) when using a structure matcher algorithm for all generated materials.

To determine novelty, we compare unique generated structures against all training-set structures with the same reduced formula. If a structure’s reduced formula does not appear in the reference dataset, it is considered Compositionally novel. If a generated structure shares a reduced formula with a reference structure but the crystal structures do not match, it is considered Structurally novel. This matcher does not establish novelty in a broader materials-discovery sense, but serves as a practical screening tool.

Pymatgen’s StructureMatcher is used to match two chemical structures together. Unless otherwise specified, the default matcher settings are used, namely a fractional length tolerance of 0.2, a site tolerance of 0.3 Å and an angle tolerance of 5∘5^\circ. For normalisation purposes, structures are reduced to their primitive cell and scaled to equivalent volume before matching.

Prediction models and DFT validation

High-throughput property estimation for generated structures was achieved using surrogate models based on the ALIGNN architecture [65]. In § 1.1 C 1, PBE band gaps [66] were predicted using the standard “mp_gappbe_alignn” model trained on Materials Project data. For the specific requirements of § 1.1 D, we trained distinct surrogate models to predict higher-fidelity properties. HSE06 band gaps [67] were predicted using an ALIGNN model trained on ∼10k\sim10\text{k} labelled structures from Kim et al. [44], with hyperparameters according to Choudhary et al. [65]. SLMEs were predicted using a model trained on the ∼5.3k\sim5.3\text{k} entry MP SLME dataset [48].

The thermodynamic stability of generated materials was first evaluated using the MACE-MP calculator to determine the energy above the convex hull, denoted here as EhullMACEE_{\mathrm{hull}}^{\mathrm{MACE}}. MACE-MP is a graph-based foundation model employing higher-order equivariant message passing, pre-trained on MP relaxation trajectories [68]. Predicted energies were processed using the MaterialsProject2020Compatibility scheme to ensure consistency between GGA and GGA+U calculations relative to the MP dataset.

Given the model’s reported mean absolute error (MAE) of 0.057 eV/atom on the Matbench Discovery benchmark [69], stability thresholds were adjusted to account for statistical uncertainty. For the general validation in § 1.1 C 1, a threshold of EhullMACE≤0.157E_{\mathrm{hull}}^{\mathrm{MACE}} \leq0.157 eV/atom was employed, expanding upon the typical 0.1 eV/atom metastability limit [55]. Conversely, for the SLME Study in § 1.1 D, a stricter threshold of EhullMACE≤0.1E_{\mathrm{hull}}^{\mathrm{MACE}} \leq0.1 eV/atom was applied to aggressively funnel the discovery pipeline. The final VSUN metric integrates this stability assessment with checks for Validity, Uniqueness and Novelty [20].

To validate the stability of promising candidates identified in the SLME study, structure optimisation and energy calculations were performed at the DFT level to obtain EPBEhullE_{\mathrm{PBE}}^{\mathrm{hull}}. We used the Vienna abab initio Simulation Package (VASP) [70, 71], employing PBE exchange-correlation functionals and Projected Augmented Wave (PAW) potentials [72, 73] matching MP v54 selections. The relaxation protocol involved sequential optimisation of ion positions and lattice parameters, followed by a final refinement step to ensure convergence within standard MP thresholds. Total energies were derived from a final static calculation with a dense kk-point mesh, ensuring energetic consistency with reference compounds for convex hull analysis. All relaxations were performed without symmetry constraints.

Detailed optical properties were subsequently computed for verified structures using a hybrid functional workflow. Following a PBE relaxation, frequency-dependent dielectric functions were calculated using the HSE06 functional [67] within the Independent Phase Approximation (IPA). Input files were generated using Atomate2 [74], but for the final dielectric calculations, kk-point meshes were made twice as dense as those generated by Atomate2 in each reciprocal space direction to ensure convergence of optical properties. This produced a mesh with ∼2000/n\sim2000/n kk-points for a unit cell with nn atoms. Dielectric functions were converted to absorption spectra using Kramers-Kronig transformations and used to determine the SLME following the method of Yu and Zunger [47] for an absorber at 300 K and with thickness d=500d = 500 nm as is widely used [47, 75].

Supply chain sustainability was evaluated using the Herfindahl-Hirschman Index (HHI) for elemental production (HHIPHHI_P) and reserves (HHIRHHI_R), based on data from Gaultois et al. [76]. Material-level scores were computed as the mass-fraction weighted average of the constituent elements’ HHI values extracted with SMACT [77]. To prioritise candidates with diverse and secure global supply chains, we employed a composite sustainability metric defined by the Euclidean distance from the origin in the HHI space:

HHIdist=HHIP2+HHIR2(6)HHI_{\mathrm{dist}} = \sqrt{HHI_P^2 + HHI_R^2} \tag*{(6)}

A lower HHIdistHHI_{\mathrm{dist}} corresponds to reduced geopolitical influence and higher supply security, serving as a selection criterion for the final candidate pool.

Performance Comparison for Conditional Models

When a model is evaluated on its capability to generate materials with targeted properties, different evaluation metrics are used for comparative analysis. Common statistical metrics, which include the Mean Average Error (MAEMAE), standard error (SESE), Pearson’s rr, correlation coefficient (R2R^2) and the dependent tt-test for paired samples (tt, pp), are not reiterated here. The Hit-Rate is defined in this work as the number of materials that have a predicted property that falls within a specified range of the target value, and it is determined as:

\mathrm{Hit\mathchar`-Rate}_{\varepsilon} = \frac{1}{N}\sum_{i=1}^{N} \left(\lvert\hat{y}_i-y_i\rvert\le\varepsilon\right), \tag*{(7)}

where ε=0.5\varepsilon= 0.5 eV was used for Section II C 1 and ε=1.0\varepsilon= 1.0 g/cm3^{3} for Section II C 2.

Data Availability

Processed datasets and trained models used in this work are stored on Hugging Face [78]. Data preparation, training and metric calculation workflows are all detailed in the repository notebooks [79].

Code Availability

The code for CrystaLLM-π\pi is available at the model repository [80], which includes a detailed README.md on how to load and generate structures with the models mentioned in this work, as well as how to adapt the framework for other use cases. Importantly, the Prefix attention method is called “PKV” in the repository, and the Residual attention method is defined as “Slider” in the repository. Exact training and generation hyperparameters for every experiment carried out in this work can be found in the repository configuration files [59]. The pre-built containerized API [81] and web-app interface are hosted thanks to PSDI [82] (Physical Sciences Data Infrastructure) resources and tools.

Acknowledgements

KTB acknowledges funding from EPSRC (EP/Y014405/1, EP/Y000552/1, EP/Y028775/1 and EP/Y028759/1). We acknowledge the use of University College London (UCL) Myriad High Performance Computing Facility (Myriad@UCL) and associated support services in the completion of this work. We acknowledge Young HPC access, which is partially funded by EPSRC (EP/T022213/1, EP/W032260/1 and EP/P020194/1). CB and MW are funded by a UCL start-up package. Part of this publication is based upon work from EUMINE COST Action CA21143, for the STMS supported by COST (European Cooperation in Science and Technology). KL is supported by the Bezos Earth Fund via the Earth Rover Program (G-2023-201305339). BAAM was funded by the “AI for Science” PSDI sub-project (“Provision of ‘AI ready’ data: prototyping data pipelines and repositories”, grant award UKRI2697).

Author Contributions

CB and KTB conceptualised the project and wrote the initial manuscript, the package was developed by CB as a development from the CrystaLLM code written by LA, the Residual model was based on code provided by KL. Experimental XRD data was provided by AA and JD. DFT calculations and FGW distance evaluations were performed by MW. BM helped develop the API tools and the front-end for the web-app. RG-C, LA, BAAM and KL provided feedback and guidance on project design and were also involved in editing the final manuscript.

Competing Interests

The authors declare no competing interests.

Appendices

Property-labelled Dataset Construction Details

The LeMaterial dataset does not contain any functional properties; it is exclusively made up of unlabeled CIFs derived from the source material [36], randomly split into a train:validation ratio of 90:10. The dataset was accessed in April 2025.

For property-labelled datasets, we first consider the Alex-MP-20 data [20], accessed April 2025. Structures have at most 20 atoms per unit cell and an EhullPBEE_{\mathrm{hull}}^{\mathrm{PBE}} of at most 0.1 eV/atom. All structures are relaxed with DFT using a PBE functional with corrections based on the Materials Project [55] for energy consistency. All MatterGen datasets were randomly split into train:validation sets with a 90:10 ratio. Without any property labels, this is the Alex-MP-20 dataset. The MatterGen Density Subset considered for the Dataset Size study was curated by calculating density using pymatgen’s Structure.density in (g/cm3^3). EhullPBEE_{\mathrm{hull}}^{\mathrm{PBE}} values were taken directly from the base dataset. Density and EhullPBEE_{\mathrm{hull}}^{\mathrm{PBE}} were normalised using Min-Max method. To generate XRD spectra for the conditioning vector in the MatterGen XRD dataset, XRD patterns were calculated with pymatgen’s XRDCalculator at the CuKα\alpha wavelength. The 20 most intense Bragg peaks and their 2θ2\theta positions were stored. The intensities and 2θ2\theta values were normalised with Min-Max normalisation where 0:90 were used for 2θ2\theta and 0:100 were used for min-max ranges. Patterns with under 20 Bragg peaks were padded with [−100][-100] tokens, indicating a heterogeneous “missing value”.

The Materials Project API was queried to create the MP Bandgap dataset [55], accessed April 2025. All structures are relaxed with DFT using a PBE functional with the Materials Project corrections. Band gaps and EhullPBEE_{\mathrm{hull}}^{\mathrm{PBE}} were retrieved directly from the API. Materials with null band gaps were removed. The database was randomly split into train and validation sets in a 90:10 ratio.

The MP SLME dataset was curated from Walker and Butler’s dataset, accessed in September 2025 [48], created by taking overlapping materials from the Woods-Robinson et al. and Kim et al. datasets [44, 83]. The Spectroscopic Limited Maximum Efficiency (SLME) values were derived from high-fidelity absorption spectra, and band gaps were calculated with the Heyd-Scuseria-Ernzerhof (HSE) hybrid DFT functional. CIFs were then obtained from Materials Project IDs associated with each property value. For hyperparameter search, the dataset was split into a 95:5 train:validation ratio, but for the final model, the full dataset was put into the training set.

The MP-20 dataset [55] was used to curate the MP-20 XRD dataset. The dataset was accessed in April 2025. The structures are a subset of the materials project material with at most 20 atoms per unit cell. The XRD conditioning property vectors were generated with the same methodology as the MatterGen XRD Subset. Both dataset splits with train:validation:test sets of 60:20:20 were taken directly from PXRDGen [54].

The Jarvis-DFT dataset [56] was used to curate the Jarvis-DFT XRD fine-tuning subset, accessed in January 2025. All structures are relaxed with DFT using a PBE functional. The XRD conditioning property vectors were generated with the same methodology as the MatterGen XRD Subset. The database was randomly split into train:validation:test sets of 80:10:10 for hyperparameter search and then 90:10 train:test for the final model to match the benchmark.

The Chili-100K XRD dataset contains experimentally determined crystal structures sourced from Chili-100K [84], a curated and filtered inorganic subset of the COD [85]. It was accessed in April 2026 and contains around 21K materials, funnelled down to 14K after our deduplication pipeline. The XRD conditioning vectors used in this benchmark were simulated from the deposited structures using the same methodology as the MatterGen XRD Subset. The dataset was split into train:validation:test ratios of 78.6:10.7:10.7. The test set in particular was crafted to study the model’s performance in a data leakage aware manner: 500 materials present in the test set were seen during a training phase (LeMaterial, MatterGen XRD or Chili-100K train:val subsets), 500 were materials whose reduced formula was seen during a training phase but the structure was never seen (Calculated using the Structure Novelty metric from Section V E 2) and 500 materials had reduced formulas and structures never seen in any phase of training.

Token and Atom Distributions

Image not extracted

Figure 7. Distribution of token counts and atom counts for pre-processed Crystallographic Information Files (CIFs) across all fine-tuning datasets. In each panel, the top subplot shows the token-count distribution relative to the model context limit, which is 1024 tokens for standard datasets and 1536 tokens for the Chili-100K XRD dataset. The bottom subplot shows the atom-count distribution per CIF. The histograms compare conventional-cell representations used for CrystaLLM-π\pi with primitive representations used for comparison to graph diffusion models such as MatterGen [20]. Vertical lines mark the 20-atom threshold often used to restrict diffusion-model datasets, including MP-20 [55] and Alex-MP-20 [20]. Green and yellow denote structures within the context window, whereas red and orange denote structures excluded for exceeding it. Together, these distributions show that CIF tokenisation can represent many systems well beyond 20 atoms within the available context window because only the asymmetric unit is written explicitly.

Preprocessing and Batching

The base CrystaLLM-π\pi model trains to learn patterns on a large corpus of CIFs. Before this, the relevant data needs to be pre-processed into an optimised machine-readable format. In essence, preprocessing steps include CIF deduplication, augmentation and tokenisation. They follow the same core logic as the original CrystaLLM work [38].

Deduplication was performed by keeping the lowest volume per formula unit structure for identical cell composition and space group. For data augmentation, symmetry information was embedded using pymatgen’s symmetry finding with a tolerance of 0.10.1 Å and floating point numbers were all rounded to 4 decimal places. Additionally, brackets were added to the CIF, encapsulating parts that vary between crystal structures as described in § 1.4 B, like “data_[Ge4]” where text inside the brackets is considered variable and outside the brackets is considered fixed. For an example of a fully augmented CIF, see Appendix VI D.

During tokenisation, CIFs were wrapped in <bos> and <eos> to favour learning of termination behaviour and clear example boundaries. Tokenisation was performed on CIF chunks such as digits, atom types or CIF tags in the same manner as CrystaLLM [38], producing a final token vocabulary of 377. A condition vector for each material was generated by concatenating associated properties of interest when property conditioning methods were activated.

For batching, sequences longer than the context length were truncated and shorter ones were stacked in a round-robin manner until the length was reached, detailed in Pseudo-code 1. The input sequences were allowed to contain multiple CIFs; however, when conditioning, only the first CIF’s condition vector was applied. Training data was shuffled at each epoch during training, so CIF stacking was never the same twice during training.

Algorithm 1  Round-robin packing of CIF token sequences to context length $C$

Input:  List of $N$ token sequences $\{X_0, X_1, \ldots, X_{N-1}\}$ (Each $X$ is a CIF in dataloader)
Output: Packed sequence $out$ with length exactly $C$ (context length)

1:
2: function RoundRobin(features, $C$)
3:     for $i \leftarrow 0$ to $N - 1$ do
4:         $X \leftarrow$ features[$i$].$X$
5:
6:         if len($X$) $\ge C$ then                                      ▷ CIF is longer than context, slice it
7:             $out \leftarrow X[0:C]$
8:             return $out$
9:         end if
10:
11:         $out \leftarrow X$                                            ▷ Start with current CIF
12:         $current\_index \leftarrow i$
13:
14:         while len($out$) $< C$ do                                      ▷ Append more until context reached
15:             $current\_index \leftarrow (current\_index + 1) \bmod N$  ▷ Next in batch
16:             $next\_CIF \leftarrow$ features[$current\_index$].$X$
17:             $remaining\_space \leftarrow C -$ len($out$)
18:             $tokens\_to\_take \leftarrow$ min(len($next\_CIF$), $remaining\_space$)
19:             Append $next\_CIF[0:tokens\_to\_take]$ to $out$
20:         end while
21:
22:         $out \leftarrow out[0:C]$                                      ▷ trim to exact length if needed
23:         return $out$
24:     end for
25: end function

Augmented CIF

Example of an augmented CIF for Ge$_4$

Figure 8. Example of an augmented CIF for Ge4_4. Bracketed fields mark the special tokens introduced during augmentation. Variable tokens are shown in red and fixed tokens are shown in black. This representation physically separates learnable structural content from invariant CIF syntax.

LeMat-Benchmark

ModelMethodValid ↑\uparrowUnique ↑\uparrowNovel ↑\uparrowStable ↑\uparrowMetastable ↑\uparrowSUN ↑\uparrowMSUN ↑\uparrowEhullE_{\mathrm{hull}} (eV) ↓\downarrowEformE_{\mathrm{form}} (eV) ↓\downarrowRMSD (Å) ↓\downarrow
Crystal-GFN [86]GFN51.7%51.7%51.7%0.0%0.0%0.0%0.0%2.0858-1.30361.8665
LLaMat3 [87]AR15.4%15.2%10.5%0.1%2.1%0.0%0.2%1.70670.74231.0057
ADiT [88]Diff90.6%87.8%26.0%0.4%36.5%0.0%1.0%0.3333-0.73390.3794
LLaMat2 [87]AR84.4%81.4%30.0%0.7%34.7%0.1%2.1%0.4395-0.47260.5363
SymmCD [89]Diff73.4%73.0%47.0%1.4%18.6%0.1%2.4%0.8761-0.01950.8720
CrystalFormer [23]AR69.9%69.4%31.8%1.4%28.8%0.0%3.1%0.7039-0.16890.6585
Zatom-1-WD [90]Flow67.2%65.4%24.2%0.2%22.8%0.0%0.4%0.5189-0.36870.4773
CrystaLLM-π (ours)AR86.8%84.9%24.9%2.9%49.0%0.3%3.6%0.3205-0.59030.3631
DiffCSP++ [91]Diff95.3%95.1%62.0%1.0%26.4%0.2%5.0%0.4093-0.51890.6933
DiffCSP [92]Diff95.7%94.8%66.2%2.3%29.8%0.1%8.5%0.2747-0.63670.5857
Chemeleon1 [14]Diff95.4%94.8%64.0%3.6%34.4%0.3%11.3%0.2055-0.75900.4907
Chemeleon2 [93]Diff + RL95.2%88.1%71.6%0.0%39.8%0.0%21.2%0.1557-0.71670.4226
MP-20 Pre-Relaxed
WyFormer [26]AR93.4%93.0%66.4%0.5%15.7%0.1%1.9%0.4988-0.43060.8121
WyFormer-DFT [26]AR95.2%95.0%66.4%3.7%24.8%0.4%7.8%0.2708-0.66600.4173
PLaID++ [94]AR + RL96.0%77.8%24.2%12.4%60.7%1.0%7.6%0.0854-0.49750.1286
Zatom-1-WD [90]Flow88.3%86.0%37.9%4.7%50.5%0.3%8.8%0.1093-0.82020.3387
MatterGen [20]Diff95.7%95.1%70.5%2.0%33.4%0.2%15.0%0.1834-0.70200.3878
MiAD [95]Diff96.2%94.3%40.2%6.4%62.0%1.0%16.6%0.0804-0.84360.2494
OMatG [96]Diff + Flow96.4%95.2%51.2%11.6%49.8%1.0%18.0%0.0956-0.92270.0759
MCFlow [97]Flow97.2%96.3%52.2%11.9%49.3%0.7%18.9%0.0987-0.85130.1696
Crystalite [98]Diff97.2%95.8%53.2%12.7%51.6%1.5%22.6%0.0905-0.89160.1322
OMatG-FC [96]Diff + Flow97.2%92.8%28.9%18.4%57.9%1.7%12.0%0.0694-0.91490.0685
Alex-MP-20 Non-Pre-Relaxed
CrystaLLM-π (ours)AR93.4%93.2%23.9%4.8%71.3%0.3%12.5%0.0941-0.58110.1914
Chemeleon1 [14]Diff94.0%93.9%59.8%4.0%54.4%1.0%28.0%0.1120-0.63440.3089
Chemeleon2 [93]Diff + RL97.4%93.5%93.7%0.3%67.4%0.2%62.4%0.0959-0.60860.2678
Alex-MP-20 Pre-Relaxed
None

Table VI (PDF p. 21). Generative performance metrics for models evaluated on the MP-20 and Alex-MP-20 benchmarks within LeMat-Bench [99]. The table distinguishes non-pre-relaxed and pre-relaxed generation protocols, where the latter includes a structural relaxation step before candidate evaluation and the former evaluates out-of-the-box unrelaxed structures. Entries within each section are sorted by ascending cumulative Stable, Unique, Novel (SUN) and Meta-Stable, Unique, Novel (MSUN) scores (bottom row is best). Upward arrows (↑\uparrow) specify metrics where higher values indicate superior generative capacity (Valid, Unique, Novel, Stable, Metastable, SUN, MSUN). Downward arrows (↓\downarrow) indicate metrics where lower values represent greater stability and structural accuracy (EhullE_{\mathrm{hull}}, EformE_{\mathrm{form}}, RMSD). The method column denotes the model family: autoregressive (AR), diffusion (Diff), generative flow network (GFN), flow matching (Flow) and reinforcement learning (RL). Bold and italics denote the best and second-best metrics within each subdivision.

Output Logit Study

The custom digit-wise tokenisation scheme induces a hierarchical generation process for atomic coordinates. The model predicts leading integers and primary decimals with high certainty to establish coarse spatial boundaries, while subsequent trailing decimals display broader probability distributions. This enables stochastic exploration within defined structural limits.

Autoregressive logit analysis heatmaps for continuous crystallographic parameters

Figure 9. Autoregressive logit analysis for continuous crystallographic parameters in the generated Rb2(NbBr3)3\mathrm{Rb}_2(\mathrm{NbBr}_3)_3 structure sequence. Each heatmap corresponds to one parameter shown above the subplot. Columns denote digit positions in the generated token sequence, while rows denote vocabulary bins consisting of digits 0 to 9, the decimal point “.” and an aggregated “other” category. The colour scale shows the predicted probability mass P(token)P(\mathrm{token}), and black boxes mark the tokens sampled in the final Rb2(NbBr3)3\mathrm{Rb}_2(\mathrm{NbBr}_3)_3 sequence. This material was absent from both the pre-training and fine-tuning datasets (§ 1.1 D and VI S). The distributions show that the model concentrates probability within the digit space for continuous-valued fields, with early digits fixing coarse geometric constraints at high confidence and later decimal positions remaining broader to support fine structural variation.

Architectural details for Encoders

Residual Encoder and Prefix Encoder architectures. Residual Encoder: Input Conditions; Tanh + Dropout; Linear to KV size; Einsum; Linear to hidden dimension; Tanh + Dropout; Linear to KV size; Einsum; Expand x h; Split; $K_{\mathrm{Residual}}$ and $V_{\mathrm{Residual}}$. Prefix Encoder: Input Conditions; Linear to hidden dimension; Layer Norm + ReLU; Expand x $2(K+V)$; Linear to KV size; Expand x h; Dropout + Split; $K_{\mathrm{Prefix}}$ and $V_{\mathrm{Prefix}}$.

Figure 10. Residual and Prefix conditioning encoder architectures. The left panel shows the Residual encoder, and the right panel shows the Prefix encoder. Both map the input condition vector to learned key-value tensors that are replicated across the NN decoder blocks of the transformer. The Residual encoder uses einsum-based expansions so that individual input properties occupy distinct regions of the latent space and can be associated with separate key-value pairs. The Prefix encoder instead learns a shared set of key-value tensors that is not directly attributable to individual input properties. This difference mirrors the downstream attention design, where Residual attention preserves property-specific structure and Prefix attention injects a shared conditioning context.

Architectural details for PrependGPT

Prepend conditioning architecture diagram showing the Prepend Encoder, GPT-2 Backbone, and GPT-2 Attention. Labels include Input Conditions, Linear to hidden dimension, Layer Norm + ReLU, Linear to Input Embedding size, Condition Embeddings, concat, Input Embeddings, Positional Encoding, Layer Norm, Linear to QKV size, $Q_{\mathrm{seq}}$, $K_{\mathrm{seq}}$, $V_{\mathrm{seq}}$, M-H Attention, Feed Forward, Linear & Softmax, MHA output, Scaled Dot-Product Attention, $\times h$, and $\times N$.

Figure 11. Prepend conditioning architecture. The Prepend encoder maps the input condition vector to the same embedding dimension as the input sequence embeddings. These condition embeddings are concatenated with the token embeddings before entering the GPT-2 backbone. The downstream attention mechanism then remains unchanged relative to the standard GPT-2 block. This makes Prepend conditioning a sequence-level intervention rather than an attention-level modification.

Experimental protocols

Structures for the LeMat-Bench evaluation were generated using models pre-trained on either the standard MP-20 or Alex-MP-20 datasets (§ 1.1 A). Following benchmark requirements, 2.5K CIFs were sampled. For the MP-20 dataset, sampling parameters were set to T=1.0T = 1.0, top-K=50K = 50 and top-p=0.95p = 0.95. For the Alex-MP-20 dataset, parameters were adjusted to T=1.25T = 1.25, top-K=50K = 50 and top-p=0.95p = 0.95. Ab initio prompting was employed uniformly without specifying target cell compositions.

The MP Bandgap dataset was used for model training in § 1.1 C 1. All four conditioning architectures were evaluated to assess information retention and baseline conditioning capabilities. Models were either trained from scratch or fine-tuned from the pre-trained backbone. Sampling across both initialisation regimes generated 1K CIFs per condition using T=0.7T = 0.7, top-K=30K = 30 and top-p=0.95p = 0.95, with ab initio prompting. To evaluate performance across regions of varying training data density, eight target band gap conditions were tested: 0.07, 0.18, 0.68, 1.55, 2.44, 3.72, 6.78 and 12.1 eV (inputted as normalised values). A secondary condition of Ehull=0E_{\mathrm{hull}} = 0 eV/atom was applied uniformly across all trials.

For § 1.1 C 2, the MatterGen Density dataset was randomly sampled to produce subsets containing 1K, 10K and 100K structures (Figure 12). Pre-trained models were fine-tuned on each subset to evaluate conditional generation performance under constrained data regimes.

Density distributions of the original MatterGen density dataset and its randomly sampled subsets

Figure 12. Density distributions of the original MatterGen density dataset and its randomly sampled subsets. The histogram shows the frequency of material densities (g/cm³) on a logarithmic scale for the full dataset together with random subsets of 100K, 10K and 1K entries. Smaller subsets progressively lose coverage in the high-density tails, illustrating how limited random samples can underrepresent the full target space.

Sampling generated 1K CIFs per condition using T=1.0T = 1.0, top-K=15K = 15 and top-p=0.95p = 0.95, with ab initio prompting. Eight density conditions were evaluated: 1.27, 2.55, 6.37, 10.20, 12.75, 15.30, 19.12 and 22.94 g/cm³ (inputted as normalised values), with Ehull=0E_{\mathrm{hull}} = 0 eV/atom applied as a secondary condition.

For the discovery task, the complete MP SLME dataset was used to fine-tune the base model for SLME-conditioned CIF generation. Sampling produced 100K CIFs using T=1.0T = 1.0, top-K=50K = 50 and top-p=0.95p = 0.95, with ab initio prompting. The elevated top-KK parameter expanded the accessible chemical space during atomic species selection, preventing collapse toward common elements and increasing the elemental diversity of the generated structures.

Materials recovery tasks used the XRD datasets. Given the highly constrained nature of these tasks, like incorporating multiple conditions, heterogeneous data, cell compositions and occasionally space group information, a “1-perplexity” sampling scheme was employed for crystal structure predictions (ablation detailed in Appendix VI § 1.5.14). Sampling parameters were fixed at T=0.75T = 0.75, top-K=10K = 10 and top-p=0.95p = 0.95. The temperature parameter was determined via a systematic sweep that optimised match rates and RMSD metrics on a sub-sampled test set for both “20-consistent” and “1-perplexity” scored CIFs (see § 1.4 D for sampling scheme and Appendix VI § 1.5.14 for the ablation study). All models were prompted with target cell compositions alongside XRD spectra. For the polymorph recovery study (Section III E 2), we used T=1.1T = 1.1, top-K=15K = 15 and top-p=0.95p = 0.95. Models were additionally evaluated with space group prompting and supercell prompting by doubling the input stoichiometry relative to the true cell composition.

Pre-training Study Extras

Parity distributions comparing ALIGNN-predicted band gaps of generated materials against requested targets

Figure 13. Parity distributions comparing ALIGNN-predicted band gaps of generated materials against requested targets across conditioning architectures and training regimes. Columns distinguish the from scratch and pre-trained learning regimes, while rows correspond to the Prefix, Residual, Prepend and Raw conditioning models. Box plots summarise the distributions of predicted band gaps for QVSUNQ_{\mathrm{VSUN}} structures at discrete, non-linearly spaced target values. The dashed diagonal line marks perfect calibration between requested and generated values. The grey background profile shows the band gap distribution of the training dataset. Panel (h) is empty because the Raw baseline failed to generate Valid structures after pre-training. Together with the quantitative results in Table VII, these distributions indicate that the pre-trained Prefix architecture provides the most accurate and consistent target tracking across the sampled property range.

Target-space analysis of conditioning mechanisms with respect to pre-training initialisation

Figure 14. Target-space analysis of conditioning mechanisms with respect to pre-training initialisation. The panels show (a) target hit rate, (b) fraction of structurally valid generations and (c) fraction of generations satisfying the comprehensive QVSUNQ_{\mathrm{VSUN}} criterion (Validity, Stability, Uniqueness and Novelty; Methods V E) across specific band gap targets, with 1K generation attempts per target (Appendix VII). Colours denote the Residual, Prefix, Prepend and Raw models. Circles and solid lines indicate pre-trained models, whereas triangles and dashed lines indicate models trained from scratch. The grey background shows the reference training-set band gap density. As targets move farther out of distribution, hit rate declines more rapidly than structural validity or QVSUNQ_{\mathrm{VSUN}}, indicating that the models preserve physically plausible generation more readily than exact numerical agreement at extreme targets. The Raw token-level baseline is included only as a failure reference because it does not generate Valid structures in the pre-trained regime.

Dataset Size Study extras

SizeModelNValidN_{\mathrm{Valid}}R2R^2MAE (± SE)Valid-Hit (%)Q-Hit (%)
1KPrefix6681-0.276.42 (0.096)8.263.15
1KResidual7170-0.697.68 (0.096)6.342.69
1KPrepend5914-0.537.35 (0.104)5.262.45
10KPrefix7618-0.125.97 (0.081)10.384.59
10KResidual7526-0.617.44 (0.091)6.443.05
10KPrepend7609-0.657.56 (0.090)6.552.97
100KPrefix77000.831.98 (0.039)36.9312.40
100KResidual77340.772.35 (0.046)34.6113.05
100KPrepend77350.782.39 (0.045)33.4512.28
FullPrefix76790.970.72 (0.016)56.7016.32
FullResidual77450.951.00 (0.021)49.3316.14
FullPrepend76860.960.84 (0.019)55.4316.51

Table VIII. Density-conditioning performance across fine-tuning dataset sizes and architectures. NValidN_{\mathrm{Valid}} is the total number of structurally valid generations. Valid-Hit is the percentage of those within ±1.0 g/cm3\pm1.0\ \mathrm{g/cm^3} of the requested target, and Q-Hit is the percentage of those generations that also satisfy the QVSUNQ_{\mathrm{VSUN}} criteria. R2R^2 and MAE ±\pm SE are computed between requested and predicted densities for QVSUNQ_{\mathrm{VSUN}} materials only. Bold and italics denote the best and second-best results within each dataset size.

Comparison with graph diffusion

Figures 15, 17 and 16 show behavioral differences between MatterGen and CrystaLLM-π\pi. Figure 16 highlights a marked difference in compositional complexity between the CrystaLLM-π\pi and MatterGen [20]. The former more often produces unary to senary compositions, whereas the graph diffusion baseline sometimes generates structures with substantially larger numbers of distinct elements, reaching up to 31 in a single unit cell in this benchmark. One likely contributor is that the autoregressive CIF representation encodes stoichiometry through discrete tokens, while the graph diffusion representation does not impose the same explicit compositional scaffold during generation. Because convex-hull analysis scales poorly for very large chemical systems, these were intractable on available hardware, so structures with more than six unique elements were excluded from the stability calculations.

Output-space distribution of band-gap-conditioned generations for CrystaLLM-$\pi$ Prefix and MatterGen

Figure 16. Output-space distribution of band gap-conditioned generations comparing the from scratch CrystaLLM-π\pi Prefix architecture with the MatterGen graph diffusion baseline. The histogram reports counts across generated band gap bins, where translucent bars denote structurally valid generations and solid bars denote the subset satisfying the comprehensive QVSUNQ_{\mathrm{VSUN}} metric (Validity, Stability, Uniqueness and Structural Novelty; Methods V E). Dashed vertical lines mark the requested target band gaps, and the shaded grey background shows the training-set band gap distribution. No method generated structures satisfying the 12.1 eV target, which explains its absence in the output space. These distributions show that MatterGen produces more Valid and QVSUNQ_{\mathrm{VSUN}} structures across most of the sampled range, while both methods fail at the most extreme target.

CrystaLLM-π\pi also yields substantially higher symmetry retention in the unrelaxed outputs considered here. Because the model generates discrete crystallographic fields hierarchically (Appendix VI F), many outputs remain compatible with higher-order space-group assignments before post-processing. By contrast, the unrelaxed MatterGen outputs are strongly biased toward P1 in this analysis, consistent with small coordinate deviations obscuring symmetry before relaxation. At symprec = 0.2 Å, approximately 69% of MatterGen structures still fail to recover higher-order symmetry, compared with 0.2% for CrystaLLM-π\pi. This comparison helps explain why relaxation is particularly important when evaluating continuous-coordinate diffusion outputs.

Finally, Table IX shows a large computational gap between the two frameworks in this benchmark. CrystaLLM-π\pi was run on a 24G L4 GPU with a peak VRAM demand of 8.99 GB, whereas the MatterGen baseline required 40.35 GB on 40G A100 GPUs. Training and inference were also substantially faster for the autoregressive model under the reported settings. These differences are consistent with the lower sampling complexity of the 1D autoregressive pipeline relative to graph-based diffusion. In this setting, the discrete token approach therefore offers a lighter route to conditional generation, although the trade-off with structural yield remains as discussed in § 1.1 C 1.

ModelGPUTrain Time (HH:MM:SS)Inference Time (1K Gens)Max VRAM (GB)
CrystaLLM-π\piL4 (24G)03:47:4700:02:268.99
MatterGenA100 (40G)325:30:0615:06:0840.35

Table IX. Computational resource requirements and processing times for CrystaLLM-π\pi and MatterGen on the MP Bandgap dataset. In the evaluation set, 95% of structures contain between 1 and 104 atoms per unit cell. Reported metrics are the hardware used, peak video random access memory (VRAM), total training duration and inference time required to generate 1K structures. MatterGen [20] was trained for 1100 epochs across an unconditional Phase 1 stage and a conditional Phase 2 stage with a classifier-free guidance adapter. CrystaLLM-π\pi was trained for 24 epochs.

Parity distributions comparing ALIGNN-predicted band gaps for CrystaLLM-$\pi$ Prefix and MatterGen

Figure 15. Parity distributions comparing ALIGNN-predicted band gaps against requested target values for materials generated by the CrystaLLM-π\pi Prefix model and the MatterGen graph diffusion baseline. Panel a shows the from scratch CrystaLLM-π\pi Prefix model and panel b shows the from scratch MatterGen baseline. Box plots summarise the distributions of predicted band gaps for QVSUNQ_{\mathrm{VSUN}} generations at discrete, non-linearly spaced target values. The dashed diagonal line marks ideal calibration between requested and generated values. The shaded grey background profile shows the band gap distribution of the training dataset. These distributions indicate that the CrystaLLM-π\pi Prefix architecture maintains tighter calibration and narrower spread across most of the sampled property range than the graph diffusion baseline.

Distributional analysis of structural parameters for 8K materials generated by the MatterGen baseline and the from scratch CrystaLLM-$\pi$ Prefix architecture across eight target band gaps

Figure 17. Distributional analysis of structural parameters for 8K materials generated by the MatterGen baseline and the from scratch CrystaLLM-π\pi Prefix architecture across eight target band gaps. Panel a shows the probability density of the total atom count per unit cell. Panel b shows the fraction of generated structures as a function of compositional complexity, where the terminal 7+ bin collects all outliers, including MatterGen generations with up to 31 distinct elements. Panels c through i summarise symmetry outcomes as crystal-system pie charts: panel c reports the reference training-set distribution at symprec = 0.1 Å, panels d, e, and f report CrystaLLM-π\pi Prefix generations at symprec = 0.01, 0.01 and 0.2 Å, and panels g, h, and i report MatterGen generations at the same symprec values. In these pies, P1 is treated as its own category, separate from triclinic. The shaded grey background in panels a and b shows reference distributions extracted from the training dataset. Together, these comparisons show that CrystaLLM-π\pi Prefix tracks the training-set distributions more closely in atom count, compositional complexity and crystal-system assignment, whereas MatterGen produces more high-complexity compositions and remains strongly biased toward P1 even as symprec increases.

Losses

Evolution of the total training loss across the transition from unconditional pre-training to conditional fine-tuning

Figure 18. Evolution of the total training loss across the transition from unconditional pre-training to conditional fine-tuning on the MP Bandgap dataset. The horizontal axis uses a broken scale to show the initial pre-training descent (left) together with the late-stage convergence and fine-tuning transition near 1.25×1061.25 \times10^{6} trainer steps (right). The solid grey curve traces the unconditional pre-training loss. The Raw, Residual, Prepend and Prefix conditioning architectures were then fine-tuned for up to 50K additional steps with matched hyperparameters and early stopping. These trajectories show that the Residual architecture undergoes the least disruption at the transition, whereas the Raw method destabilises strongly and does not recover the pre-training loss level.

Evolution of individual training loss components for the generative model trained from scratch on the MP-20 XRD benchmark

Figure 19. Evolution of individual training loss components for the generative model trained from scratch on the MP-20 XRD benchmark. The plot shows the total train loss, language-modelling (LM) loss and format loss evaluated with component weighting {n1,n2}={1,1}\{n_{1}, n_{2}\} = \{1, 1\}. The format loss decays to near zero within the first 1K steps, after which the total loss is dominated by the LM term. This indicates that the model learns the invariant Crystallographic Information File (CIF) syntax early and then spends most of its training on the variable structural content.

Sampling Scheme Parameter Studies for Structure Recovery Tasks

To determine optimal hyperparameters and generation strategy for the crystal structure recovery tasks, a series of ablation studies were conducted on a 1K structure subset of the MP-20 test set.

The first study, shown in Table X, gives a sampling temperature of T=0.75T = 0.75 as a good balance between recovery accuracy and generative diversity. While a lower temperature of T=0.5T = 0.5 yields a marginally better RMSD, selecting T=0.75T = 0.75 increases the 1-perplexity match rate by 1.4% points, maximising the probability of identifying the correct structure in a single closed loop attempt. Higher temperatures offer a slight increase in the match rate for 20-consistent CIF generation attempts, but at the cost of reduced structural accuracy.

ModelTempRanking SchemeMatch Rate (%)RMSDAvg. Perplexity
CrystaLLM-π\pi0.520-consistent73.20.03791.151
0.7520-consistent75.30.03931.164
1.020-consistent76.40.04671.172
1.2520-consistent76.30.04611.176
1.520-consistent76.60.05321.181
1.720-consistent76.20.05801.181
0.51-perplexity63.60.04081.136
0.751-perplexity65.00.04651.148
1.01-perplexity63.20.05041.154
1.251-perplexity61.00.04731.154
1.51-perplexity58.40.05111.158
1.71-perplexity56.60.05111.156

Table X. Ablation study for sampling-temperature optimisation. The model was trained from scratch on the MP-20 XRD dataset. “NN-consistent” denotes generation until NN self-consistent CIFs are obtained, where self-consistent means passing basic chemical validity checks. “1-perplexity” denotes selecting the lowest-perplexity output among those NN-consistent CIFs for matching, as described in § 1.4 D.

The second study, shown in Table XI, confirms the benefits of both XRD conditioning and perplexity-based ranking. Providing XRD information consistently improves both the 1-perplexity match rate (+3.3% points) and the RMSD (−0.0058). Then, ranking candidates by perplexity provides an improvement again for both metrics without any trade-off.

ModelTempRanking SchemeXRD InfoMatch Rate (%)RMSDAvg. Perplexity
CrystaLLM-π\pi0.7520-shotNo73.80.0427−
0.7520-shotYes75.30.0418−
0.7520-consistentYes75.30.03931.164
0.751-shotNo59.50.0589−
0.751-shotYes62.80.0531−
0.751-perplexityYes65.00.04651.148

Table XI. Ablation study on the utility of XRD conditioning and perplexity-based ranking. The model was trained from scratch on the MP-20 XRD dataset. “NN-shot” denotes matching against NN generated candidates. “NN-consistent” denotes generation until NN self-consistent CIFs are obtained, where self-consistent means passing basic chemical validity checks. “1-perplexity” denotes selecting the lowest-perplexity output among those NN-consistent CIFs for matching, as described in § 1.4 D. All models were trained with XRD information, and the XRD ablation removes that information only at inference time.

Data Leakage Analysis for XRD Structure Recovery

Large-scale pre-training on the 4.35 million structures introduces the risk of data leakage when evaluating downstream tasks. To decouple the model’s intrinsic structure-solving capabilities from potential memorisation of the pre-training data, the novelty of each material in the test set was defined against the union of the pre-training (LeMaterial) and fine-tuning (Jarvis-DFT train/validation) datasets.

  1. Seen: Materials where both the exact structural configuration and the reduced formula are present in the reference data using pymatgen’s StructureMatcher with the VSUN metric parameters (See § 1.4 E 2)

  1. Novel Polymorph (Novel Polym.): Materials where the reduced formula exists in the reference data, but the specific structural polymorph is absent.

  1. Novel Composition (Novel Comp.): Materials where the reduced formula and structure is completely absent from all reference datasets.

Under the “20-consistent” evaluation scheme, the match rate increases from 84.99% for “Seen” materials to 87.85% for “Novel Compositions”, while the RMSD improves from 0.0360 A˚\mathring{\mathrm{A}} to 0.0288 A˚\mathring{\mathrm{A}}. A plausible interpretation is that memorised structure priors can sometimes compete with the conditioning signal for “Seen” compositions, whereas this competition is weaker for novel compositions. However, the split is also affected by sample composition: although the average atom counts are similar (16.5 for “Seen” versus 15.9 for “Novel Compositions”), the “Seen” subset contains larger outliers, reaching 232 atoms compared with 116 for novel targets. The observed advantage for novel compositions should therefore be interpreted as dataset-specific rather than as a general rule that out-of-distribution recovery is easier.

ModelNum gensSubsetN in bucketMatch Rate (%)RMSD (A˚\mathring{\mathrm{A}})MAE a (A˚\mathring{\mathrm{A}})MAE b (A˚\mathring{\mathrm{A}})MAE c (A˚\mathring{\mathrm{A}})MAE V (A˚3\mathring{\mathrm{A}}^3)Avg. AtomsMax Atoms
CrystaLLM-π\pi20Seen425184.990.03600.1310.1310.2169.35916.5232
CrystaLLM-π\pi20Novel Polym.271685.680.03800.1580.1370.21611.96015.6160
CrystaLLM-π\pi20Novel Comp.60987.850.02880.1420.1500.21323.12515.9116
CrystaLLM-π\pi1Seen425166.670.03430.0960.0970.1869.01317.0232
CrystaLLM-π\pi1Novel Polym.271665.460.03730.1040.1060.1887.10016.1160
CrystaLLM-π\pi1Novel Comp.60966.500.02600.0620.0670.1407.47417.3116

Table XII. Stratified performance metrics on the Jarvis-DFT benchmark for evaluating data leakage. The test set is split by strict novelty criteria relative to the combined pre-training and fine-tuning datasets. Structural fidelity is assessed by root mean square displacement (RMSD), mean absolute errors (MAE) of lattice parameters and cell volume, and the average and maximum atom counts of successfully recovered structures. The “20” and “1” “Num gens” columns correspond to the “20-consistent” and “1-perplexity” generation settings, respectively.

Default Hyperparameters for Pre-training and Framework Validation

The model backbone consists of 26 million parameters with an embedding size of 512, 8 decoder layers and 8 attention heads, following CrystaLLM’s small model architecture [38]. Training employed DeepSpeed Stage 2 optimisation across 2 NVIDIA L4 GPUs with early stopping based on validation loss plateau or increase to avoid overfitting. For the exact hyperparameter values used during training, see Table XIII and configuration files are available in the project repository [59].

Hyperparameterpre-trainingfine-tuning
OptimiserAdamWAdamW
Learning Rate10−310^{-3}5×10−65 \times10^{-6}
LR ScheduleCos. to 0.1%LRCos. to 1% LR
β1\beta_{1}0.850.9
β2\beta_{2}0.980.999
Gradient Clipping1.01.0
Batch Size3232
Dropout0.10.1
Weight Decay0.10.01
Context Length10241024
Training Steps1.250MVaries
Warmup0 steps2% of steps
Cond. Hidden Dim.–1024
Cond. Dropout–0.01
Cond. LR–5×10−45 \times10^{-4}
Cond. Weight Decay–0.01

Table XIII. Hyperparameter settings for the pre-training and conditional fine-tuning studies spanning 50K samples (pre-training benefits) and 1.4K–400K samples (dataset-size study). From-scratch models used a backbone LR of 10−310^{-3}, a conditional LR of 10−310^{-3}, weight decay of 0.1 and 15K training steps. “Cond.” denotes values specific to the conditioning encoder. LR denotes learning rate. “Cos.” denotes a cosine-annealed learning-rate schedule to a target fraction of the base learning rate.

Unless hyperparameter searches were carried out, default training hyperparameters were used in the various studies of this work. For default pre-training on the structure-only dataset, the model used cosine learning rate scheduling with the AdamW [60] optimiser and pre-training on the text-only MP-20 and Alex-MP-20 for the LeMat-Benchmark (Section VI E) used the Muon [100] optimiser. Conditional fine-tuning continued from pre-trained checkpoints with reduced learning rates for backbone parameters, while newly initialised conditional encoders used higher learning rates. The conditional encoder hidden dimension was set to 1024 with reduced regularisation compared to pre-training.

Models trained from scratch in comparative studies used default fine-tuning hyperparameters with higher regularisation. Training duration varied by study design, with pre-training benefits experiments using 50K steps and dataset size studies scaling from 1.4K to 400K steps based on dataset size. Exact configuration files for all training and fine-tuning runs are available in the model repository [80].

Discovery and Recovery hyperparameters

Hyperparameter optimisation for specialised conditional generation tasks employed either Optuna with a MedianPruner [101] or a Weights & Biases (WandB) Bayesian sweep across 50 trials. The optimisation objective targeted the minimisation of validation loss. Validation sets were generated by withholding 5% of the data for SLME studies and 10% for the Chili-100K and Jarvis dataset. Early stopping prevented overfitting during both the search and final training. Exact configuration files for all optimised training and fine-tuning runs are available in the model repository [80].

DatasetArchitectureInitializationHoldoutTrialsOptimized Parameters
MP SLMEPrefixPre-trained5%50Cond. Prefix Tokens: {1,2,3,4,5}\{1, 2, 3, 4, 5\}; Cond. Hidden Dim.: {512,1024}\{512, 1024\}; Base LR: log⁡U(5×10−7,5×10−5)\log U(5 \times10^{-7}, 5 \times10^{-5}); Cond. Dropout: U(0.0,0.2)U(0.0, 0.2), Warmup steps: U(50,500)U(50, 500); Base WD: U(0.01,0.2)U(0.01, 0.2), Cond. WD: U(0.01,0.3)U(0.01, 0.3)
MP-20 XRDResidualScratchExisting50Base LR: log⁡U(1×10−5,1×10−2)\log U(1 \times10^{-5}, 1 \times10^{-2}); Cond. Dropout: U(0.0,0.2)U(0.0, 0.2), Warmup: U(0.01,0.10)U(0.01, 0.10); Base WD: U(0.01,0.2)U(0.01, 0.2), Cond. WD: U(0.01,0.3)U(0.01, 0.3); Backbone Dropout: U(0.0,0.2)U(0.0, 0.2)
Jarvis XRDResidualPre-trained10%50Base LR: log⁡U(5×10−7,5×10−5)\log U(5 \times10^{-7}, 5 \times10^{-5}); Cond. LR: log⁡U(1×10−4,1×10−3)\log U(1 \times10^{-4}, 1 \times10^{-3}); Warmup: U(0.01,0.10)U(0.01, 0.10), Cond. Dropout: U(0.0,0.2)U(0.0, 0.2); Base WD: U(0.01,0.2)U(0.01, 0.2), Cond. WD: U(0.01,0.3)U(0.01, 0.3)
Chili-100K XRDResidualSequential10%50Base LR: log⁡U(5×10−4,5×10−3)\log U(5 \times10^{-4}, 5 \times10^{-3}); Cond. LR: log⁡U(5×10−3,5×10−2)\log U(5 \times10^{-3}, 5 \times10^{-2}); Warmup Ratio: U(0.01,0.10)U(0.01, 0.10), Cond. Dropout: U(0.0,0.2)U(0.0, 0.2); Base WD: U(0.01,0.1)U(0.01, 0.1), Cond. WD: U(0.01,0.1)U(0.01, 0.1)

Table XIV. Hyperparameter search spaces for specialised conditional generation across multiple materials datasets. Continuous distributions are denoted by UU (uniform) and log⁡U\log U (log-uniform), while discrete options are enclosed in braces. Validation loss optimisation used Optuna (MedianPruner, TPESampler) for all studies except Chili-100K, which used a Weights & Biases Bayesian sampler, and all searches were limited to 50 trials. Total training steps were SLME (16K), MP-20 scratch (30K) and Jarvis (16K). The Chili-100K XRD models underwent sequential fine-tuning with an initial Phase 1 adaptation on MatterGen XRD (500K steps) followed by Phase 2 Chili-100K optimisation (2.5K steps). Abbreviations: LR (learning rate), WD (weight decay), Cond. (conditioning encoder parameters).

Open Sourced Models

Model NameStudyAvailabilityModel ParametersTraining DatasetMethodProperty Vector
CrystalLLM-π\pi BaseII AOpen-Source25.9MLeMaterialBaseNone
CrystalLLM-π\pi MP-20II AOpen-Source25.9MMP-20BaseNone
CrystalLLM-π\pi Alex-MP-20II AOpen-Source25.9MAlex-MP-20BaseNone
CrystalLLM-π\pi Band gapII C 1Open-Source61.6MMP BandgapPrefix[Band gap, EhullPBEE_{\mathrm{hull}}^{\mathrm{PBE}}]
CrystalLLM-π\pi DensityII C 2Open-Source61.6MMatterGen DensityPrefix[Density, EhullPBEE_{\mathrm{hull}}^{\mathrm{PBE}}]
CrystalLLM-π\pi Chili-100K-XRDII E 2Open-Source47.7MChili-100K XRDResidual[20×2θ20 \times2\theta, 20×intensity20 \times\text{intensity}]
CrystalLLM-π\pi SLMEII DOpen-Source38.7MMP SLMEPrefix[SLME]

Table XV. Overview of open-source CrystalLLM-π\pi model variants. The table summarises the model releases associated with this work, including their training datasets, parameter counts and the conditioning architecture used. Although additional trained models are available upon request, this table lists the public releases for the main studies in the paper. The models can be found in the paper’s Hugging Face repository [102].

SLME Candidate Structures

References

References

  1. [1]D. Antypov, A. Vasylenko, C. Collins, L. Daniels, G. Darling, M. Dyer, J. Claridge, and M. Rossinsky, Discovery of crystalline inorganic solids in the digital age, Accounts of chemical research 58, 1355 (2025).DOI
  2. [2]A. Merchant, S. Batzer, S. S. Schoenholz, M. Aykol, G. Cheon, and E. D. Cubuk, Scaling deep learning for materials discovery, Nature 624, 80 (2023).DOI
  3. [3]D. W. Davies, K. T. Butler, A. J. Jackson, A. Morris, J. M. Frost, J. M. Skelton, and A. Walsh, Computational screening of all stoichiometric inorganic materials, Chem 1, 617 (2016).pubmed.ncbi.nlm.nih.gov/27790643
  4. [4]F. Tiwari, Exploring the World of Functional Materials Revolutionizing Technology and Beyond, Advanced Materials Science Research 6, 1 (2023).
  5. [5]U. Nwabara, K. Yang, A. Talekar, V. Bernales, J. González, S. Miller, and J. Wu, High throughput computational and experimental methods for accelerated electrochemical materials discovery, Journal of Materials Chemistry A 13, 26041 (2025).DOI
  6. [6]Z. Allahyari and A. R. Oganov, Coevolutionary search for optimal materials in the space of all possible compounds, npj Computational Materials 6, 55 (2020).DOI
  7. [7]K. T. Butler, D. W. Davies, H. Cartwright, O. Isayev, and A. Walsh, Machine learning for molecular and materials science, Nature 559, 547 (2018).DOI
  8. [8]G. Hautier, Finding the needle in the haystack: Materials discovery and design through computational ab initio high-throughput screening, Computational Materials Science 163, 108 (2019).DOI
  9. [9]J. Armstrong, A. J. O'Malley, M. R. Ryder, and K. T. Butler, Understanding dynamic properties of materials using neutron spectroscopy and atomistic simulation, Journal of Physics Communications 4, 072001 (2020).DOI
  10. [10]H. Park, Z. Li, and A. Walsh, Has generative artificial intelligence solved inverse materials design?, Matter 7, 2355 (2024).DOI
  11. [11]B. Sanchez-Lengeling and A. Aspuru-Guzik, Inverse molecular design using machine learning: Generative models for matter engineering, Science 361, 360 (2018).DOI
  12. [12]P.-P. De Breuck, H.-C. Wang, G.-M. Rignanese, S. Botti, and M. A. Marques, Generative ai for crystal structures: A review, arXiv preprint arXiv:2509.02723 (2025).arxiv.org/abs/2509.02723
  13. [13]T. Xie, X. Fu, O.-E. Ganea, R. Barzilay, and T. Jaakkola, Crystal Diffusion Variational Autoencoder for Periodic Material Generation (2022), arXiv:2110.06197 [cs].arxiv.org/abs/2110.06197
  14. [14]H. Park, A. Onwuli, and A. Walsh, Exploration of crystal chemical space using text-guided generative artificial intelligence (2024).DOI
  15. [15]N. Gruver, A. Sriram, A. Madotto, A. G. Wilson, C. L. Zitnick, and Z. W. Ulissi, Fine-tuned language models generate stable inorganic materials as text, in International Conference on Learning Representations 2024 (2024).arxiv.org/abs/2402.04379
  16. [16]B. K. Miller, R. T. Q. Chen, A. Sriram, and B. M. Wood, FlowMM: Generating Materials with Riemannian Flow Matching (2024), arXiv:2406.04713 [cs].arxiv.org/abs/2406.04713
  17. [17]T. Pakornchote, N. Choomphon-anomakhun, S. Arrerut, C. Atthapak, S. Khamkaeo, T. Chotibut, and T. Bovornratanaraks, Diffusion probabilistic models enhance variational autoencoder for crystal structure generative modeling (2023), arXiv:2308.02165 [cs].arxiv.org/abs/2308.02165
  18. [18]C.-Y. Ye, H.-M. Weng, and Q.-S. Wu, Con-CDVAE: A method for the conditional generation of crystal structures, Computational Materials Today 1, 100003 (2024).DOI
  19. [19]X. Luo, Z. Wang, P. Gao, J. Lv, Y. Wang, C. Chen, and Y. Ma, Deep learning generative model for crystal structure prediction, npj Computational Materials 10, 254 (2024).DOI
  20. [20]C. Zeni, R. Pinsler, D. Zügner, A. Fowler, M. Horton, X. Fu, Z. Wang, A. Shysheya, J. Crabbé, S. Ueda, R. Sordillo, L. Sun, J. Smith, B. Nguyen, H. Schulz, S. Lewis, C.-W. Huang, Z. Lu, Y. Zhou, H. Yang, H. Hao, J. Li, C. Yang, W. Li, R. Tomioka, and T. Xie, A generative model for inorganic materials design, Nature 639, 1 (2025).DOI
  21. [21]S. Yang, K. Cho, A. Merchant, P. Abbeel, D. Schuurmans, I. Mordatch, and E. D. Cubuk, Scalable Diffusion for Materials Generation (2024), arXiv:2311.09235 [cs].arxiv.org/abs/2311.09235
  22. [22]K. Yan, X. Li, H. Ling, K. Ashen, C. Edwards, R. Arroyave, M. Zitnik, H. Ji, X. Qian, X. Qian, and S. Ji, Invariant Tokenization of Crystalline Materials for Language Model Enabled Generation, in The Thirty-eighth Annual Conference on Neural Information Processing Systems (2024).arxiv.org/abs/2503.00152
  23. [23]Z. Cao, X. Luo, J. Lv, and L. Wang, Space Group Informed Transformer for Crystalline Materials Generation, https://arxiv.org/abs/2403.15734v2 (2024).
  24. [24]T. Mohanty, M. Mehta, H. M. Sayeed, B. Oded, I. Pitussi, A. Borenstein, V. Srikumar, and T. D. Sparks, CrysText: A Generative AI Approach for Text-Conditioned Crystal Structure Generation using LLM (2025).DOI
  25. [25]P.-P. D. Breuck, H. A. Piracha, G.-M. Rignanese, and M. A. L. Marques, A generative material transformer using Wyckoff representation (2025), arXiv:2501.16051 [cond-mat].arxiv.org/abs/2501.16051
  26. [26]N. Kazeev, W. Nong, I. Romanov, R. Zhu, A. Ustyuazhanin, S. Yamazaki, and K. Hippalgaonkar, Wyckoff transformer: Generation of symmetric crystals, URL https://arxiv. org/abs/2503.02407 (2025).arxiv.org/abs/2503.02407
  27. [27]N. Alampara, S. Miret, and K. M. Jablonka, Matttext: Do language models need more than text & scale for materials modeling?, arXiv preprint arXiv:2406.17295 (2024).
  28. [28]A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, Attention Is All You Need (2023), arXiv:1706.03762.
  29. [29]K. Choudhary, AtomGPT: Atomistic Generative Pretrained Transformer for Forward and Inverse Materials Design, The Journal of Physical Chemistry Letters 15, 6909 (2024).DOI
  30. [30]Y. Xia, P. Jin, S. Xie, L. He, C. Cao, R. Luo, G. Liu, Y. Wang, Z. Liu, Y.-J. Chen, Z. Guo, Y. Bai, P. Deng, Y. Min, Z. Lu, H. Hao, H. Yang, J. Li, C. Liu, J. Zhang, J. Zhu, K. Wu, W. Zhang, K. Gao, Q. Pei, Q. Wang, X. Liu, Y. Li, H. Zhu, Y. Lu, M. Ma, Z. Wang, T. Xie, K. Maziarz, M. Segler, Z. Yang, Z. Chen, Y. Shi, S. Zheng, L. Wu, C. Hu, P. Dai, T.-Y. Liu, H. Liu, and T. Qin, NatureLM: Deciphering the Language of Nature for Scientific Discovery (2025), arXiv:2502.07527 [cs].arxiv.org/abs/2502.07527
  31. [31]K. Choudhary, DiffractGPT: Atomic Structure Determination from X-ray Diffraction Patterns using Generative Pre-trained Transformer (2024).
  32. [32]K. M. Jablonka, P. Schwaller, A. Ortega-Guerrero, and B. Smit, Leveraging Large Language Models for Predictive Chemistry (2023).DOI
  33. [33]A. A. Levy and M. Geva, Language Models Encode Numbers Using Digit Representations in Base 10 (2025), arXiv:2410.11781 [cs].arxiv.org/abs/2410.11781
  34. [34]H. Li, X. Chen, Z. Xu, D. Li, N. Hu, F. Teng, Y. Li, L. Qiu, C. J. Zhang, Q. Li, and L. Chen, Exposing Numeracy Gaps: A Benchmark to Evaluate Fundamental Numerical Abilities in Large Language Models (2025), arXiv:2502.11075 [cs].arxiv.org/abs/2502.11075
  35. [35]J. Schmidt, T. F. Cerqueira, A. H. Romero, A. Loew, F. Jäger, H.-C. Wang, S. Botti, and M. A. Marques, Improving machine-learning models in materials science through large datasets, Materials Today Physics 48, 101560 (2024).
  36. [36]Martin Siron, Nel Djafar, Lucile Ritchie, Etienne Du-Fayet, Amandine Rossello, Ali Ramlaoui, Leandro von Werra, Thomas Wolf, and Alexandre Duval, Delmat-bulkunique dataset (2024).
  37. [37]S. Kirklin, J. E. Saal, B. Meredig, A. Thompson, J. W. Doak, M. Aykol, S. Rühl, and C. Wolverton, The Open Quantum Materials Database (OQMD): Assessing the accuracy of DFT formation energies, npj Computational Materials 1, 15010 (2015).DOI
  38. [38]L. M. Antunes, K. T. Butler, and R. Grau-Crespo, Crystal structure generation with autoregressive large language modeling, Nature Communications 15, 10570 (2024).arxiv.org/abs/2307.04340
  39. [39]A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, Language Models are Unsupervised Multitask Learners, Tech. Rep. (OpenAI, 2019).
  40. [40]M. McCloskey and N. J. Cohen, Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem, in Psychology of Learning and Motivation, Vol. 24, edited by G. H. Bower (Academic Press, 1989) pp. 109–165.DOI
  41. [41]X. L. Li and P. Liang, Prefix-Tuning: Optimizing Continuous Prompts for Generation (2021), arXiv:2101.00190 [cs].arxiv.org/abs/2101.00190
  42. [42]E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, LoRA: Low-Rank Adaptation of Large Language Models (2021), arXiv:2106.09685 [cs].arxiv.org/abs/2106.09685
  43. [43]M. Piacentini, D. W. Lynch, and C. Olson, Thermoreflectance of lif between 12 and 30 ev, Physical Review B 13, 5530 (1976).DOI
  44. [44]S. Kim, M. Lee, C. Hong, Y. Yoon, H. An, D. Lee, W. Jeong, D. Yoo, Y. Kang, Y. Youn, and S. Han, A band-gap database for semiconducting inorganic materials calculated with hybrid functional, Scientific Data 7, 387 (2020).DOI
  45. [45]S. Park, B. Lee, S. H. Jeon, and S. Han, Hybrid functional study on structural and electronic properties of oxides, Current Applied Physics 11, S337 (2011).DOI
  46. [46]W. Shockley and H. J. Queisser, Detailed Balance Limit of Efficiency of p-n Junction Solar Cells, Journal of Applied Physics 32, 510 (1961).
  47. [47]L. Yu and A. Zunger, Identification of Potential Photovoltaic Absorbers Based on First-Principles Spectroscopic Screening of Materials, Physical Review Letters 108, 068701 (2012).pubmed.ncbi.nlm.nih.gov/22401127
  48. [48]M. Walker and K. T. Butler, The carbon cost of materials discovery: Can machine learning really accelerate the discovery of new photovoltaics? (2025), arXiv:2507.13246 [cond-mat].arxiv.org/abs/2507.13246
  49. [49]J. H. Song, H. Choi, H. T. Pham, and S. Jeong, Energy level tuned indium arsenide colloidal quantum dot films for efficient photovoltaics, Nature Communications 9, 4267 (2018).DOI
  50. [50]N. Papež, R. Dallaev, Š. Tälu, and J. Kaštyl, Overview of the Current State of Gallium Arsenide-Based Solar Cells, Materials 14, 3075 (2021).DOI
  51. [51]J. Dias, M. Andrade, and L. Mascaro, Chapter 13 - chalcogenide compounds for solar cells, in Green Sustainable Process for Chemical and Environmental Engineering and Science, edited by Inamuddin, R. Boddula, A. M. Asiri, and M. M. Rahman (Elsevier, 2021) pp. 245–270.
  52. [52]K. Kuhar, A. Crovetto, M. Pandey, K. S. Thygesen, B. Seger, P. C. Vesborg, O. Hansen, I. Chorkendorff, and K. W. Jacobsen, Sulfide perovskites for solar energy conversion applications: computational screening and synthesis of the selected compound lays 3, Energy & Environmental Science 10, 2579 (2017).DOI
  53. [53]S. Lu, H. Lin, L. Yao, Z. Gao, X. Ji, W. E. L. Zhang, and G. Ke, Uni-3DAR: Unified 3D Generation and Understanding via Autoregression on Compressed Spatial Tokens (2025), arXiv:2503.16278 [cs].
  54. [54]Q. Li, R. Jiao, L. Wu, T. Zhu, W. Huang, S. Jin, Y. Liu, H. Weng, and X. Chen, Powder diffraction crystal structure determination using generative models, Nature Communications 16, 7428 (2025).arxiv.org/abs/2409.04727
  55. [55]A. Jain, S. P. Ong, G. Hautier, W. Chen, W. D. Richards, S. Dacek, S. Cholia, D. Gunter, D. Skinner, G. Ceder, and K. A. Persson, Commentary: The Materials Project: A materials genome approach to accelerating materials innovation, APL Materials 1, 011002 (2013).DOI
  56. [56]K. Choudhary, K. F. Garrity, A. C. E. Reid, B. DeCost, A. J. Biacchi, A. R. H. Walker, Z. Trautt, J. Hattrick-Simpers, A. G. Kusne, A. Centrone, A. Davydov, J. Jiang, R. Pachter, G. Cheon, E. Reed, A. Agrawal, X. Qian, V. Sharma, H. Zhuang, S. V. Kalinin, B. G. Sumpter, G. Pilania, P. Acar, S. Mandal, K. Haule, D. Vanderbilt, K. Rabe, and F. Tavazza, The Joint Automated Repository for Various Integrated Simulations (JARVIS) for data-driven materials design, npj Computational Materials 6, 10.1038/s41524-020-00440-1 (2020), arXiv:2007.01831 [cond-mat].DOI
  57. [57]S. Golkar, M. Pettee, M. Eickenberg, A. Bietti, M. Cranmer, G. Krawczik, F. Lanusse, M. McCabe, R. Ohana, L. Parker, B. R.-S. Blancard, T. Tesileanu, K. Cho, and S. Ho, xval: A continuous numerical tokenization for scientific language models (2024), arXiv:2310.02989 [stat.ML].arxiv.org/abs/2310.02989
  58. [58]C. Bone, Hugging Face Datasets: C-Bone Datasets, https://huggingface.co/c-bone/datasets (2026), accessed: 2026-05-18.
  59. [59]C. Bone, CrystalLLM-pi Configuration Files (Tag v1.3.0), https://github.com/C-Bone-UCL/CrystalLLM-pi/tree/main/config_files (2026), accessed: 2026-05-18.
  60. [60]I. Loshchilov and F. Hutter, Decoupled Weight Decay Regularization (2019), arXiv:1711.05101 [cs].arxiv.org/abs/1711.05101
  61. [61]J. Howard and S. Ruder, Universal Language Model Fine-tuning for Text Classification, https://arxiv.org/abs/1801.06146v5 (2018).
  62. [62]A. Fan, M. Lewis, and Y. Dauphin, Hierarchical Neural Story Generation (2018), arXiv:1805.04833 [cs].arxiv.org/abs/1805.04833
  63. [63]A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi, The Curious Case of Neural Text Degeneration, in International Conference on Learning Representations (2019).arxiv.org/abs/1904.09751
  64. [64]S. P. Ong, W. D. Richards, A. Jain, G. Hautier, M. Kocher, S. Cholia, D. Gunter, V. L. Chevrier, K. A. Persson, and G. Ceder, Python Materials Genomics (pymatgen): A robust, open-source python library for materials analysis, Computational Materials Science 68, 314 (2013).DOI
  65. [65]K. Choudhary and B. DeCost, Atomistic Line Graph Neural Network for improved materials property predictions, npj Computational Materials 7, 185 (2021).DOI
  66. [66]J. P. Perdew, K. Burke, and M. Ernzerhof, Generalized Gradient Approximation Made Simple, Physical Review Letters 77, 3865 (1996).pubmed.ncbi.nlm.nih.gov/10062328
  67. [67]J. Heyd, G. E. Scuseria, and M. Ernzerhof, Hybrid functionals based on a screened Coulomb potential, The Journal of Chemical Physics 118, 8207 (2003), _eprint: https://pubs.aip.org/aip/jcp/article-pdf/11/18/8207/19093575/8207_1_online.pdf.
  68. [68]I. Batatia, P. Benner, Y. Chiang, A. M. Elena, D. P. Kovács, J. Riebesell, X. R. Advincula, M. Asta, M. Avaylon, W. J. Baldwin, F. Berger, N. Bernstein, A. Bhowmik, S. M. Blau, V. Cárare, J. P. Darby, S. De, F. D. Pia, V. L. Deringer, R. Elijošius, Z. El-Machachi, F. Falcioni, E. Fako, A. C. Ferrari, A. Genreith-Schriever, J. George, R. E. A. Goodall, C. P. Grey, P. Grigorev, S. Han, W. Handley, H. H. Heenen, K. Hermansson, C. Holm, J. Jaafar, S. Hofmann, K. S. Jakob, H. Jung, V. Kapil, A. D. Kaplan, N. Karimitari, J. R. Kermode, N. Kroupa, J. Kullgren, M. C. Kuner, D. Kurylia, G. Liepuoniute, J. T. Margraf, I.-B. Magdău, A. Michaelides, J. H. Moore, A. A. Naik, S. P. Niblett, S. W. Norwood, N. O’Neill, C. Ortner, K. A. Persson, K. Reuter, A. S. Rosen, L. E. Schaaf, C. Schran, B. X. Shi, E. Sivonxay, T. K. Stenczel, W. Svahn, C. Sutton, T. D. Swinburne, J. Tilly, C. van der Oord, E. Varga-Umbrich, T. Vegge, M. Vondrák, Y. Wang, W. C. Witt, F. Zills, and G. Csányi, A foundation model for atomistic materials chemistry (2024), arXiv:2401.00096 [physics].arxiv.org/abs/2401.00096
  69. [69]J. Riebesell, R. E. A. Goodall, P. Benner, Y. Chiang, B. Deng, G. Ceder, M. Asta, A. A. Lee, A. Jain, and K. A. Persson, A framework to evaluate machine learning crystal stability predictions, Nature Machine Intelligence 7, 836 (2025).
  70. [70]G. Kresse and J. Hafner, Ab initio molecular dynamics for liquid metals, Physical Review B 47, 558 (1993).pubmed.ncbi.nlm.nih.gov/10004490
  71. [71]G. Kresse and J. Furthmüller, Efficiency of ab-initio total energy calculations for metals and semiconductors using a plane-wave basis set, Computational Materials Science 6, 15 (1996).
  72. [72]G. Kresse and J. Hafner, Norm-conserving and ultrasoft pseudopotentials for first-row and transition elements, Journal of Physics: Condensed Matter 6, 8245 (1994).
  73. [73]G. Kresse and D. Joubert, From ultrasoft pseudopotentials to the projector augmented-wave method, Phys. Rev. B 59, 1758 (1999).
  74. [74]A. M. Ganose, H. Sahasrabuddhe, M. Asta, K. Beck, T. Biswas, A. Bonkowski, J. Bustamante, X. Chen, Y. Chiang, D. C. Chrzan, J. Clary, O. A. Cohen, C. Ertural, M. C. Gallant, J. George, S. Gerits, R. E. A. Goodall, R. D. Guha, G. Hautier, M. Horton, T. J. Inizan, A. D. Kaplan, R. S. Kingsbury, M. C. Kuner, B. Li, X. Linn, M. J. McDermott, R. S. Mohanakrishnan, A. N. Naik, J. B. Neaton, S. M. Parmar, K. A. Persson, G. Petretto, T. A. R. Purcell, F. Ricci, B. Rich, J. Riebesell, G.-M. Rignanese, A. S. Rosen, M. Scheffler, J. Schmidt, J.-X. Shen, A. Sobolev, R. Sundararaman, C. Tezak, V. Trinquet, J. B. Varley, D. Vigil-Fowler, D. Wang, D. Waroquiers, M. Wen, H. Yang, H. Zheng, J. Zheng, Z. Zhu, and A. Jain, Atomate2: modular workflows for materials science, Digital Discovery 4, 1944 (2025).DOI
  75. [75]D. H. Fabini, M. Koerner, and R. Seshadri, Candidate inorganic photovoltaic materials from electronic structure-based optical absorption and charge transport proxies, Chemistry of Materials 31, 1561 (2019).
  76. [76]M. W. Gaultois, T. D. Sparks, C. K. H. Borg, R. Seshadri, W. D. Bonificio, and D. R. Clarke, Data-Driven Review of Thermoelectric Materials: Performance and Resource Considerations, Chemistry of Materials 25, 2911 (2013).
  77. [77]D. W. Davies, K. T. Butler, A. J. Jackson, J. M. Shelton, K. Morita, and A. Walsh, Smact: Semiconducting materials by analogy and chemical theory, Journal of Open Source Software 4, 1361 (2019).DOI
  78. [78]C. Bone, Hugging Face Profile: C-Bone, https://huggingface.co/c-bone (2026), accessed: 2026-05-18.
  79. [79]C. Bone, CrystalLLM-pi Repository Notebooks (Tag v1.3.0), https://github.com/C-Bone-UCL/CrystalLLM-pi/tree/main/notebooks (2026), accessed: 2026-05-18.
  80. [80]C. Bone, CrystalLLM-pi Model Repository (Tag v1.3.0), https://github.com/C-Bone-UCL/CrystalLLM-pi (2026), accessed: 2026-05-18.
  81. [81]C. Bone, Containerized API, https://resources.psdi.ac.uk/resource-theme/37242589-7d89-462a-afad-29386bcfcda4 (2026), accessed: 2026-05-18.
  82. [82]Physical Sciences Data Infrastructure, PDSI, https://www.psdi.ac.uk/ (2026), accessed: 2026-05-18.
  83. [83]R. Woods-Robinson, Y. Xiong, J.-X. Shen, N. Winner, M. K. Horton, M. Asta, A. M. Ganose, G. Hautier, and K. A. Persson, Designing transparent conductors using forbidden optical transitions (2023), arXiv:2303.16994 [cond-mat].
  84. [84]U. Friis-Jensen, F. L. Johansen, A. S. Anker, E. B. Dam, K. M. O. Jensen, and R. Selvan, Chili: Chemically-informed large-scale inorganic nanomaterials dataset for advancing graph machine learning (Association for Computing Machinery, New York, NY, USA, 2024) p. 4962–4973.
  85. [85]S. Gražulis, D. Chateigner, R. T. Downs, A. F. T. Yokochi, M. Quirós, L. Lutterotti, E. Manakova, J. Butkus, P. Moeck, and A. Le Bail, Crystallography Open Database – an open-access collection of crystal structures, Journal of Applied Crystallography 42, 726 (2009).
  86. [86]M. Al4Science, ; A. Hernandéz-Garcia, A. Duval, A. Volokhova, Y. Bengio, D. Sharma, P. L. Carrier, Y. Benabed, M. Koziarski, V. Schmidt, G.-M. Rignanese, P.-P. D. Breuck, and P. Clancy, Crystal-gfn: sampling crystals with desirable properties and constraints (2026), arXiv:2310.04925 [cs.LG].
  87. [87]V. Mishra, S. M. Singh, D. Ahlawat, M. Zaki, V. Bihani, H. S. Grover, B. Mishra, S. Miret, Mausam, and N. M. A. Krishnan, Foundational large language models for materials research (2025), arXiv:2412.09560 [cond-mat.mtrl-sci].
  88. [88]C. K. Joshi, X. Fu, Y.-L. Liao, V. Gharakhanyan, B. K. Miller, A. Sriram, and Z. W. Ulissi, All-Atom diffusion transformers: Unified generative modelling of molecules and materials, in Forty-second International Conference on Machine Learning (2025).
  89. [89]D. Levy, S. S. Panigrahi, S.-O. Kaba, Q. Zhu, K. L. K. Lee, M. Galkin, S. Miret, and S. Ravanbakhsh, SymmCD: Symmetry-preserving crystal generation with diffusion models, in The Thirteenth International Conference on Learning Representations (2025).
  90. [90]A. Morehead, M. Cretu, A. Panescu, R. Anand, M. Weiler, T. Perez, S. Blaus, S. Farrell, W. Bhimji, A. Jain, H. Sahasrabuddhe, P. Lio, T. Jaakkola, R. Gomez-Bombarelli, R. Ying, N. B. Erichson, and M. W. Mahoney, Zatom-1: A multimodal flow foundation model for 3d molecules and materials (2026), arXiv:2602.22251 [cs.LG].arxiv.org/abs/2602.22251
  91. [91]R. Jiao, W. Huang, Y. Liu, D. Zhao, and Y. Liu, Space group constrained crystal generation, in The Twelfth International Conference on Learning Representations (2024).arxiv.org/abs/2402.03992
  92. [92]R. Jiao, W. Huang, P. Lin, J. Han, P. Chen, Y. Lu, and Y. Liu, Crystal structure prediction by joint equivariant diffusion, arXiv preprint arXiv:2309.04475 (2023).arxiv.org/abs/2309.04475
  93. [93]H. Park and A. Walsh, Guiding generative models to uncover diverse and novel crystals via reinforcement learning (2025), arXiv:2511.07158 [cs.LG].arxiv.org/abs/2511.07158
  94. [94]A. Xu, R. Desai, L. Wang, G. Hope, and E. Ritz, Plaid++: A preference aligned language model for targeted inorganic materials design (2025), arXiv:2509.07150 [cs.LG].arxiv.org/abs/2509.07150
  95. [95]A. Okhotin, M. Nakhodnov, N. Kazeev, A. E. Ustyuzhanin, and D. Vetrov, Miad: Mirage atom diffusion for de novo crystal generation (2025), arXiv:2511.14426 [cs.LG].arxiv.org/abs/2511.14426
  96. [96]P. Höllmuller, T. Egg, M. Martinossyan, E. Fuemmeler, Z. Shui, A. Gupta, P. Prakash, A. Roitberg, M. Liu, G. Karypis, M. Transtrum, R. Hennig, E. B. Tadmor, and S. Martiniani, Open materials generation with stochastic interpolants, in Forty-second International Conference on Machine Learning (2025).arxiv.org/abs/2502.02582
  97. [97]K. Seong, S. Ahn, S. Han, and C. Park, Multimodal crystal flow: Any-to-any modality generation for unified crystal modeling (2026), arXiv:2602.20210 [cs.LG].arxiv.org/abs/2602.20210
  98. [98]T. H. Veljković, J. Rosenthal, I. Lončarić, and J.-W. van de Meent, Crystalite: A lightweight transformer for efficient crystal modeling (2026), arXiv:2604.02270 [cs.LG].arxiv.org/abs/2604.02270
  99. [99]S. Betala, S. P. Reddi, A. Ramlouai, A. Xu, G. Channing, D. Levy, C. Fourrier, N. Kazeev, C. K. Joshi, S.-O. Kaba, F. Therrien, A. Hernandez-Garcia, R. Mercado, N. M. A. Krishnan, and A. Duval, Lemat-genbench: A unified evaluation framework for crystal generative models, arXiv preprint arXiv:2512.04562 (2025).arxiv.org/abs/2512.04562
  100. [100]K. Jordan, Y. Jin, V. Boza, Y. Jiacheng, F. Cesis ta, L. Newhouse, and J. Bernstein, Muon: An optimizer for hidden layers in neural networks (2024).
  101. [101]T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama, Optuna: A Next-generation Hyperparameter Optimization Framework (2019), arXiv:1907.10902 [cs, stat].arxiv.org/abs/1907.10902
  102. [102]C. Bone, Hugging Face Repository: C-Bone Models, https://huggingface.co/c-bone/models (2026), accessed: 2026-05-18.

Paper details

vlnc:W_2hzn9m48
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Physics and Astronomy

Not recorded for this paper

View statement
Looking for related papers…
Contents