WorksPhysics and Astronomy
Crystal Structure Based Graph Neural Network And LLM Multi-Modal Approach for Material Magnetic Ordering Prediction
Abstract
AI/ML models are useful for the prediction of materials' properties. Recently, there has been great progress in developing Neural Network models to predict the properties of solids, molecules, and crystals, avoiding expensive and time-consuming laboratory testing. This study presents a novel AI/ML-based multi-modal approach to predict material magnetic ordering using material crystal structure information as input to a Graph Neural Network model (GNN) and semantic text descriptions of the material's properties and structure as input to a Large Language Model (LLM), before performing information fusion of the GNN and LLM outputs to predict material magnetic ordering in four classes (Non-magnetic, Ferromagnetic, Antiferromagnetic, Ferrimagnetic). The model is trained, validated, and tested on a dataset which is a combination of Materials Project dataset, Magndata dataset, and a set of uranium compounds with magnetic ordering determined in laboratory testing, with an 80/10/10 (training/validation/testing) data split. The proposed GNN and LLM multi-modal approach achieves better overall accuracy than both the standalone GNN and LLM approaches with three fusion strategies.
Introduction
Material properties are critical to material engineering and determine where materials can be applied. For example, the energy band gap can be used to determine electricity conductivity and optical properties, the formation energy can be used to determine material stability, and magnetic order can indicate material performance under different thermal, mechanical, or electrical conditions. This information is then used to determine if the material can be used for different applications such as shielding, sensors, and actuators.
Experimental determination of material properties, while accurate, can be very time-consuming and costly. To mitigate this problem, Density Functional Theory (DFT) is proposed in [1, 2, 3] as a way to mathematically model the quantum mechanical behavior of atoms and electrons, which can be used to mathematically predict material properties. Though this mathematical approach is very popular, its prediction accuracy is heavily dependent on known approximation functions, and it cannot simulate complicated compound structure accurately.
Recent research has focused on the integration of AI/ML based approaches into this field, and there has been much progress in developing Graph Neural Network models to accurately predict properties of solids, molecules and crystals, avoiding traditional laboratory testing. Schütt et al. [4] created SchNet, one of the first influential GNNs capable of modeling local interactions in the molecule without requiring atomic positions to lie on a fixed grid (a challenge faced by convolution neural networks (CNN)). These interactions were captured using interaction blocks that aggregated information from neighboring atoms and would be weighted atom-wise to produce a final prediction. Xie and Grossman [5] introduced the Crystal Graph Convolutional Neural Network (CGCNN) which adapted the previous framework onto crystalline compounds, encoding atoms as nodes and bonds as edges in a crystal graph and producing predictions that could match the accuracy of DFT-computed properties across a broad range of crystal types. Louis et al. [6] introduced the Global Attention Graph Neural Network (GATGNN) model, which differentiates the contributions of individual atoms further by introducing a global attention layer and augmenting graph-attention layers and incorporates temperature, pressure, and entropy as global state inputs. In 2021, Choudhary and DeCost [7] introduced the Atomistic Line Graph Neural Network (ALIGNN) model, one of the first models to effectively incorporate bond angle data as input, which achieved top prediction precision at time of proposal.
Since the introduction of transformer architecture, the huge success of transformer-based large language models (LLMs) proposed by Vaswani et al. [8] to process language-based problems demonstrates the potential of utilizing a transformer-based approach to tackle problems in the domain of material properties prediction. Yan et al. [9] proposed a transformer designed specifically for crystalline materials, employing invariant geometric descriptors of Euclidean distances and angles and equivariant vector representation. Walker et al. [10] proposed MatBERT (Bidirectional Encoder Representations from Transformers (BERT)), which uses contextualized word representations and semantically rich sentence embeddings for material prediction. Rubungo et al. [11] proposed LLM-Prop, which leverages the general purpose learning capabilities of large language models (LLMs) to predict properties of crystals from their text descriptions and outperforms MatBERT. Li et al. [12] proposed a hybrid approach to use pretrained GNN and LLM models to encode material information including crystal structure and a tree-based ML regressor to perform property prediction.
In this paper, we present a Graph Neural Network (GNN) and LLM multi-modal approach based on work in [12] to perform magnetic ordering prediction on material compounds as studied in Broyles et al. [13]. Instead of using a pretrained GNN and LLM as information embedding only, a multi-modal approach is used to design the model. The model includes four blocks: GNN, LLM, information fusion, and classification, and all four blocks will be trained together. Three information fusion approaches are tested to show that information fusion provides slightly better accuracy.
Methods
Model architecture
The GNN-LLM multi-modal classifier combines crystal structure analysis via a GNN with semantic text descriptor analysis via an LLM for predicting magnetic ordering (NM/FM/AFM/FiM). The GNN used in this approach is ALIGNN, and the LLM used is GPT-2 [14]. The model architecture is shown in Figure 1. In this study, 3 fusion strategies are evaluated:

Figure 1. Architecture of ALIGNN-LLM multi-modal classifier
Early fusion: fuse serialized GNN graph and tokenized material descriptors together as input to LLM for feature computation, and feed to classification head for final prediction
Intermediate fusion: fuse GNN and LLM output features through concatenation and feed to classification head to compute the final class prediction
Gated late fusion: create a learnable gate for each class that takes in GNN and LLM output logits and balances the weights for both branches for final prediction of classes
Blocks
The implementation blocks are illustrated in Figure 2 including GNN (ALIGNN), LLM (GPT-2) and classification head. The workflow and fusion location of the three fusion approaches are illustrated.

Figure 2. Architecture of ALIGNN-LLM multi-modal classifier
GNN Block (ALIGNN)
Material structure information is first extracted in Pymatgen Structure JSON format and then encoded using Jarvis Atom functionality [15]. Then, it is converted to a graph and line graph (which is the required input format for ALIGNN) via edge and angle embedding using RBF expansion and MLP (Multi-Layer Perceptron). ALIGNN includes the following two graphs: a Graph Convolutional Network (GCN) using atoms as nodes and bonds as edges, and a line GCN using bonds as nodes and bond angle triplets as edges. In this manner, more structural information can be captured and used to predict more accurately.
The following example illustrates the dimensions of the ALIGNN GCN and line GCN using data from MP-149 (silicon) from the Materials Project database [16]:
For the GCN, atoms serve as the nodes and atomic bonds are the edges. CGCNN (Crystal Graph Convolutional Neural Network) embedding is used to represent each atom, and provides a 92-dimensional feature vector for each atom based on its properties from the periodic table. There is one node per Si atom; thus, the atom features have dimensions [2, 92]. Each atom has 28 neighbors within a cutoff of 5Å, and for each neighbor there is one directed ingoing and outgoing edge; hence, there are 56 directed bonds for each atom, and 56 edges per node for the GCN.
For the line GCN, the atom bonds act as the nodes and angle triplets act as the edges. To determine the number of angle triplets, each atom has 28 incoming and outgoing bonds, and thus the number of bonds including each atom is 784. Hence, for each bond (which includes two atoms), the number of angle triplets is ; the total is 1544 after removing duplicates for both atoms.
The graph construction (including data dimension changes) is demonstrated in Figure 3.

Figure 3. ALIGNN Graph construction
LLM Block
The Material compound is first converted to a text description based on the ChemNLP implementation in [17]. As an example, the information of MP-149 from Materials Project is converted to the following text description:
”The chemical information includes: The chemical has an atomic formula of Si with a prototype of A; Its molecular weight is 28.09 g/mol; The atomic fractions are Si: 1.0, and the atomic values X and Z are 1.9 and 14, respectively. The structure information includes: The lattice parameters are 3.85, 3.85, 3.85 with angles 60.0, 60.0, 60.0 degrees; The top K XRD peaks are found at 28.4, 47.2, 56.0, 69.0, 76.2 degrees; The material has a density of 2.313 g/cm³; The bond distances are as follows: Si-Si: 2.36, 3.85.”
In this study, GPT-2 model is used in the LLM block. During testing, the base GPT-2 model with 124.64M parameters was found to be capable of handling the size of the dataset and complexity of the classification problem.
Fusion
The three fusion workflows are shown in Figure 2 and the data fusion happens at input, feature and classification logits respectively.
Classification Block
The classification block takes the fused information from the information fusion block and performs magnet ordering prediction using MLP based classification head.
Data preparation and training
Data Preparation
A preliminary dataset consisting of 210,579 compounds was extracted from 3 sources: database.xlsx from Broyles et al. [13], Materials Project from [16], and Magndata from [18]. database.xlsx includes 1064 compounds, Magndata includes 231 compounds, and MP includes 209,284 compounds. When there is duplication on certain material compounds in different sources, the ones with mp ordering being laboratory validated are selected over material project ones.
Compounds without an indicated magnetic ordering (i.e. non-magnetic, ferromagnetic, ferrimagnetic, or antiferromagnetic) were filtered out. The remaining dataset contained 200,431 compounds, with the following distribution of magnetic orderings as shown in Table 1:
| Magnetic Ordering | Number of data entries | Proportion of complete dataset |
| All (across all 4 classes) | 200,431 | 100% |
| Non-magnetic | 119,225 | 59.48% |
| Ferromagnetic | 64,253 | 32.06% |
| Antiferromagnetic | 3,709 | 1.85% |
| Ferrimagnetic | 13,244 | 6.61% |
Table 1. Distribution of data in final dataset
As seen in Table 1, the distribution of data is unequal - there is significantly more data for the non-magnetic class than the others. For classification problems, optimal performance of AI/ML approaches requires a relatively balanced proportion of classes in the dataset. This poses a big challenge and the multi-modal approach can provide improved accuracy as shown in § 4.
Training Procedure
The dataset is split into training, validation, and testing sets with 80%, 10%, and 10% of total dataset used for each respectively. During training, the training and validation accuracy is evaluated at the end of each epoch and the model with the highest validation accuracy up to that epoch is saved. After training, the best model is tested on the testing dataset to determine its final classification accuracy. The training/validation/testing are performed on DUAL Nvidia 3090 GTX GPU.
The three models are trained for 30 epochs each. AdamW is used as the optimizer. Class-weighted Cross Entropy Loss is used as the loss function, with class weights 1.0/1.5/20.0/5.0 for NM/FM/AFM/FiM respectively (this penalizes errors in minority classes more to address unbalance in the dataset). It is found that ALIGNN requires a larger learning rate to converge compared to the other two models. The empirical values of learning rate: 0.001, 0.0001 are assigned to ALIGNN, LLM (including multi-modal).
Results and analysis
GNN Model Performance Validation
To evaluate ALIGNN performance on the problem of the paper, two others GNN models, CGCNN v2019 [5] and coGN [19], which are top performing models from the Materials Project matbench v0.1 matbench_mp_is_metal task leaderboard [20] were tested using a gcnn_keras implementation [21]. The aforementioned training/validation/testing data splits and training configuration were used. As shown in Table 2, ALIGNN shows better overall performance (F1) than the other two.
| Class | ALIGNN | CGCNN v2019 | coGN |
| Non-magnetic | 96%/88%/91.8% | 93%/92%/93% | 94%/93%/93% |
| Ferromagnetic | 74%/83%/78.2% | 79%/64%/70% | 79%/61%/69% |
| Antiferromagnetic | 32%/42%/36.3% | 16%/64%/26% | 16%/53%/24% |
| Ferrimagnetic | 54%/55%/54.5% | 47%/59%/52% | 40%/64%/49% |
Table 2. Per-class precision, recall and F1 for GNN models On Test Dataset
LLM Model Performance Validation
To evaluate GPT-2’s performance, different LLM models used in material science research are evaluated on the dataset of the paper on same training and test setting including LLM-Prop (google T5 v1.1 small encoder)[11], MatBert-base[10], AMD-LLAMA-135M[22], distilled GPT-2[23] in addition to GPT-2 used in the paper. It can be seen from Table 3 and Table 4 that GPT-2 achieves the best performance. From the model complexity Table 7, ALIGNN achieves good accuracy relative to its model size: for example, it outperforms LLMProp with much fewer parameters. For edge devices with limited computational capacity, ALIGNN can be a good choice, as it offers a good trade-off between resource requirement and model result accuracy.
| Class | LLMPROP | distilled GPT-2 |
| Non-magnetic | 91.7%/92%/92% | 93.7%/92.3%/93% |
| Ferromagnetic | 77.8%/70.8%/74% | 79%/79%/79% |
| Antiferromagnetic | 24%/34.5%/28.3% | 43.6%/47.2%/45.3% |
| Ferrimagnetic | 43%/55.3%/48.3% | 51%/58%/54.4% |
Table 3. Per-class precision, recall and F1 for LLM models On Test Dataset
| Class | GPT-2 | MatBert | AMD-LLAMA |
| Non-magnetic | 94%/94%/94% | 93%/94%/93% | 92.7%/94%/93.5% |
| Ferromagnetic | 81%/82%/81.5% | 80%/81%/80% | 79.5%/80%/79.8% |
| Antiferromagnetic | 50%/44%/46.8% | 47%/41%/44% | 37.6%/28.3%/32.3% |
| Ferrimagnetic | 57%/52%/54.4% | 59%/50%/54% | 57.6%/50.6%/44% |
Table 4. Per-class precision, recall and F1 for LLM models On Test Dataset
| GPT-2 base | ALIGNN | LLM-Prop | Matbert | AMD-Llama | distilled GPT2 | |
| Parameters | 124M | 4.3M | 36M | 110M | 130M | 88M |
Table 7. Model Complexity
Baseline Models - Performance Validation
Three models - ALIGNN, GPT-2 base, and intermediate-fusion multi-modal approaches - were selected as the baseline models for comparison. Their training and validation results are presented in Figure 4, Figure 5 and Figure 6. All three approaches converge rapidly: training and validation accuracy quickly exceed 90% and 80%, respectively, and plateau at around 30 epochs. Testing results for each model are shown in Figure 4, Figure 5 and Figure 6 with confusion matrix, precision and recall for each class.

Figure 4. ALIGNN training, validation and testing results

Figure 5. GPT-2 training, validation and testing results

Figure 6. ALIGNN GPT-2 feature fusion multi-modal training, validation and testing results
The ALIGNN-LLM multi-modal approach achieves the best overall model accuracy across all four classes, obtaining an accuracy of 88.6% compared to 83.6% from ALIGNN alone and 86.6% from LLM alone.
The per-class testing precision, recall and F1 are shown in Table 5. As demonstrated, the ALIGNN + GPT-2 intermediate-fusion multi-modal approach achieves comparable to better classification precision and recall compared to standalone ALIGNN and GPT-2 across most classes (with the exception of the AFM class). Of the four classes, AFM has the lowest classification precision and recall for each model.
| Class | ALIGNN only | GPT-2 only | Intermediate fusion multi-modal |
| Non-magnetic | 96%/88%/91.8% | 94%/94%/94.0% | 94.8%/96.1%/95.4% |
| Ferromagnetic | 74%/83%/78.2% | 81%/82%/81.5% | 83.3%/85.2%/84.2% |
| Antiferromagnetic | 32%/42%/36.3% | 50%/44%/46.8% | 46.9%/36.5%/41.1% |
| Ferrimagnetic | 54%/55%/54.5% | 57%/52%/54.4% | 64.2%/53.4%/58.3% |
Table 5. Per-class precision, recall and F1 for all three models on test dataset
Three forms of ”randomly guessing” labels are investigated: the accuracy of ”Uniform Guessing”, ”Proportional Guessing” and ”Zero Rule” can be calculated as 25%, 46.13% and 59.48% respectively (see Section A). This clearly demonstrates the strength of the models over random guessing.
Precision-Recall AUC Analysis
To analyze the model performance on test data with unbalanced classes, model prediction probability values for each class (i.e. output of the final Softmax layer) are logged during the test process. Afterwards, the precision-recall (PR) curve for each class is plotted by choosing a class (one vs rest) and computing precision and recall for this class while iterating the decision threshold from low to high. The PR curve shows the tradeoff between precision and recall. The curves for each model are shown in Figure 7. Subsequently, the Area Under Curve (AUC) is calculated to determine how well the model maintains high precision while increasing recall. A higher AUC is preferred, as having high precision and recall is more ideal than high precision or recall alone. The ALIGNN-LLM multi-modal approach shows better performance in most classes compared to the other two models.

(a) ALIGNN-only PR curve

(b) LLM-only PR curve

(c) ALIGNN-LLM feature fusion multi-modal PR curve
Figure 7. PR curves for all three models
In the multi-modal PR curve (Figure 7c), precision starts off abnormally low for AFM. This is because the decision threshold starts off very close to 1, so the model makes very few positive predictions. The low precision shows that some of the model’s most confident predictions are incorrect. This can be improved in future work through further model calibration.
Multi-Modal fusion approach performance comparison
For the three fusion approaches, the overall accuracy of the model are 88.77% for input (early) fusion and 88.6% for feature (intermediate) fusion and 88.6% for gated fusion. The per-class precision/recall/F1 are shown in Table 6.
| Class | Input (early) fusion | Feature (intermediate) fusion | Gated (late) fusion |
| Non-magnetic | 95%/96%/95.5% | 94.8%/96.1%/95.4% | 95%/95.5%/95.3% |
| Ferromagnetic | 83.2%/85.5%/84.3% | 83.3%/85.2%/84.2% | 82.3%/85.8%/84% |
| Antiferromagnetic | 44.3%/28.3%/34.6% | 46.9%/36.5%/41.1% | 54.8%/30.0%/39% |
| Ferrimagnetic | 62.7%/55.6%/59% | 64.2%/53.4%/58.3% | 62.3%/54.2%/58% |
Table 6. Per-class precision, recall and F1 for all three fusion approaches on test dataset
Model Complexity Analysis
The complexity of the model is shown in Table 7 including LLM-Prop (google T5 small v1.1 encoder) and Matbert-base.
Conclusion and Future Work
In this paper, a novel ALIGNN-LLM multi-modal approach to predict material magnetic ordering is proposed. The multi-modal design uses ALIGNN to create crystal structure embedding and LLM to create semantic text embedding, and fuses material embedding information together to perform classification. The test results show that this multi-modal approach performs better than ALIGNN-only and LLM-only approaches. For further improvement, the dataset can be balanced by including more samples (especially for the Antiferromagnetic class) when more data becomes available. Moreover, the higher performance of the multi-modal approach comes at the cost of memory usage and computational cost. Nevertheless, the results obtained in this study highlight the potential of the approach proposed herein.
This approach can also be adapted to handle regression problems or multi-properties prediction with the use of more complicated LLM models. Moreover, this approach can be applied to problems of material generation by utilizing the decoding capabilities of the LLM block - the fused output from the LLM and ALIGNN blocks (as detailed above) can be fed into the LLM decoder to generate new materials with desired properties.
Data Availability
The data used in this study are available at the references in Broyles et al. [13]. Datasets generated during the current study are available from the corresponding author on reasonable request.
Random Guessing Algorithms
Three ”random guessing” algorithms are employed as a baseline in this paper against which the ALIGNN-LLM multi-modal approach is compared: uniform guessing, proportional guessing, and the Zero Rule.
For uniform guessing, the algorithm predicts each of the four classes (NM, FM, AFM, FiM) with equal probability (i.e. chance to predict each class). The accuracy of this algorithm is .
For proportional guessing, the algorithm predicts each class with a probability equal to its prevalence in the dataset. For instance, NM samples make up of the dataset, so the algorithm will guess NM with a probability of . This algorithm has an accuracy of :
The Zero Rule algorithm guesses the class with the largest proportion of the dataset each time. For the dataset used in this study, NM makes up the largest percentage of the data at 59.48%; hence, the model guesses NM each time. The accuracy of this model is 59.48%.
References
- [1]Parr, R.G., Yang, W.: Density-Functional Theory of Atoms and Molecules. Oxford University Press, Oxford (1989) https://doi.org/10.1093/oso/9780195092769.001.0001
- [2]March, N.H.: Electron Correlation in the Solid State. ICP, London (1999) https://doi.org/10.1142/p174
- [3]Callaway, J., March, N.H.: Density Functional Methods: Theory and Applications. Solid State Physics 38, 135–221 (1984) https://doi.org/10.1016/S0081-1947(08)60313-6
- [4]Schütt, K., Kindermans, P.-J., Sauceda Felix, H.E., Chmiela, S., Tkatchenko, A., Müller, K.-R.: SchNet: A continual-filter convolutional neural network for modeling quantum interactions. In: Advances in Neural Information Processing Systems Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. 30 Curran Associates, Inc. (2017) https://proceedings.neurips.cc/paper_files/paper/2017/file/303ed4c69846ab36c2904d3ba8573050-Paper.pdf
- [5]Xie, T., Grossman, J.C.: Crystal Graph Convolutional Neural Networks for an Accurate and Interpretable Prediction of Material Properties. Phys. Rev. Lett. 120(14), 145301 (2018) https://doi.org/10.1103/PhysRevLett.120.145301
- [6]Louis, S.-Y., Zhao, Y., Nasiri, A., Wang, X., Song, Y., Liu, F., Hu, J.: Graph convolutional neural networks with global attention for improved materials property prediction. Phys. Chem. Chem. Phys. 22(32), 18141–18148 (2020) https://doi.org/10.1039/DOCP01474E
- [7]Choudhary, K., DeCost, B.: Atomistic Line Graph Neural Network for improved materials property predictions. npj Computational Materials 7, 185 (2021) https://doi.org/10.1038/s41524-021-00650-1
- [8]Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is All you Need. In: Advances in Neural Information Processing Systems Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. 30 Curran Associates, Inc. (2017) https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdfarxiv.org/abs/1706.03762
- [9]Yan, K., Fu, C., Qian, X., Qian, X., Ji, S.: Complete and Efficient Graph Transformers for Crystal Material Property Prediction. In: The Twelfth International Conference on Learning Representations (2024) https://openreview.net/forum?id=BnQY9XiRAsarxiv.org/abs/2403.11857
- [10]Walker, N., Trewartha, A., Huo, H., Lee, S., Cruse, K., Dagdelen, J., Dunn, A., Persson, K., Ceder, G., Jain, A.: The Impact of Domain-Specific Pre-Training on Named Entity Recognition Tasks in Materials Science. Available at SSRN 3950755 (2021)DOI
- [11]Rubungo, A.N., Arnold, C., Rand, B.P., Dieng, A.B.: LLM-Prop: Predicting Physical And Electronic Properties Of Crystalline Solids From Their Text Descriptions. arXiv preprint arXiv:2310.14029 https://arxiv.org/abs/2310.14029 (2023)
- [12]Li, Y., Gupta, V., Kilic, M.N.T., Choudhary, K., Wines, D., Liao, W.-K., Choudhary, A., Agrawal, A.: Hybrid-LLM-GNN: integrating large language models and graph neural networks for enhanced materials property prediction. Digital Discovery (2025)DOI
- [13]Broyles, C., Charles, W., Ran, S.: Structure-driven prediction of magnetic order in uranium compounds. Phys. Rev. Mater. 8(11), 114405 (2024) https://doi.org/10.1103/PhysRevMaterials.8.114405
- [14]gpt2 model. https://huggingface.co/openai-community/gpt2DOI
- [15]jarvis.core.atoms. https://jarvis-tools.readthedocs.io/en/master/autoapi/jarvis/core/atoms/index.html
- [16]Material Project. https://next-gen.materialsproject.org/materials
- [17]ALIGNN-BERT. https://github.com/Jonathanlyj/ALIGNN-BERT-TL-crystal
- [18]magndata cif. https://www.cryst.ehu.es/magndata/
- [19]Ruff, R., Reiser, P., Stühmer, J., Friederich, P.: Connectivity optimized nested line graph networks for crystal structures. Digital Discovery 3(3), 594–601 (2024) https://doi.org/10.1039/d4dd00018h
- [20]Dunn, A., Wang, Q., Ganose, A., Dopp, D., Jain, A.: Benchmarking materials property prediction methods: the Matbench test set and Automatminer reference algorithm. npj Computational Materials 6, 138 (2020) https://doi.org/10.1038/s41524-020-00406-3
- [21]Reiser, P., Eberhard, A., Friederich, P.: Graph neural networks in TensorFlow-Keras with RaggedTensor representation (kgcnn). Software Impacts, 100095 (2021) https://doi.org/10.1016/j.simpa.2021.100095
- [22]AMD-Llama-135M. https://huggingface.co/distilbert/distilgpt2
- [23]Distilled-GPT2. https://huggingface.co/amd/AMD-Llama-135m