betaFeedback

WorksComputer Science

Text to Band Gap: Pre-trained Language Models as Encoders for Semiconductor Band Gap Prediction

Abstract

We investigate transformer-based language models, including RoBERTa, T5, Llama-3, and MatSciBERT, for predicting the band gaps of semiconductor materials directly from textual descriptions. The inputs encode key material features, such as chemical composition, crystal system, space group, and other structural and electronic properties. Unlike shallow machine learning models, which require extensive feature engineering, or Graph Neural Networks, which rely on graph representations derived from atomic coordinates, pretrained language models can process textual inputs directly, eliminating the need for manual feature preprocessing or structure-based encoding. Material descriptions were constructed in two formats: structured strings with a consistent template and natural language narratives generated via the ChatGPT API. Each model was augmented with a custom regression head and finetuned for band gap prediction task. Language models of different architectures and parameter sizes were all able to predict band gaps from human-readable text with strong accuracy, achieving MAEs in the range of 0.25-0.33 eV, highlighting the success of this approach for scientific regression tasks. Finetuned Llama-3, with 1.2 billion parameters, achieved the highest accuracy (MAE 0.248 eV, R2 0.891). MatSciBERT, pretrained on materials science literature, reached comparable performance (MAE 0.288 eV, R2 0.871) with significantly fewer parameters (110 million), emphasizing the importance of domain-specific pretraining. Attention analysis shows that both models selectively focus on compositional and spin-related features while de-emphasizing geometric features, reflecting the difficulty of capturing spatial information from text. These results establish that pretrained language models can effectively extract complex feature-property relationships from textual material descriptions.

Introduction

The band gap of semiconductor materials is a fundamental property that directly impacts their electronic and optical behaviors. This parameter dictates crucial attributes such as conductivity, light absorption, and emission, making it essential for the performance of various electronic, optoelectronic, and photovoltaic devices. [1] Therefore, the precise prediction and control of the band gap are vital for optimizing semiconductor applications in these fields. [2]

Band gaps are determined primarily through experimental methods, with UV-visible absorption spectroscopy and photoluminescence spectroscopy being the most commonly used techniques.[3] However, these experimental methods can only measure the band gaps of synthesized materials and are not applicable to new materials designed theoretically. On the computational side, Density Functional Theory (DFT) has been the primary tool for studying the electronic structure of materials.[4, 5] While DFT can accurately simulate electronic properties such as band structures and band gaps, its high computational cost and resource-intensive nature make it less practical for high-throughput material screening, especially for complex systems.[6]

Machine learning (ML) methods have become powerful tools for addressing the computational challenges of DFT. Shallow ML models, such as Random Forest and Support Vector Regression, are commonly used to predict materials properties like band gaps based on material descriptors.[7, 8, 9] These models offer a cost-effective alternative to solving the full quantum mechanical equations, significantly reducing computational overhead. However, these models often struggle with non-numerical features, requiring extensive preprocessing to convert material properties into numerical formats. This reliance on extensive preprocessing and feature engineering not only adds complexity but also risks discarding nuanced or qualitative information, such as symmetry, bonding environments, or textual metadata, that could be valuable for accurate property prediction.[10, 11] Recent advances in deep learning, particularly Graph Neural Networks (GNNs),[12] have significantly enhanced the modeling of structure–property relationships by representing materials as atomic graphs that encode interatomic connectivity. However, these methods inherently rely on explicit structural information and require the conversion of atomic configurations into graph representations, introducing additional preprocessing steps and limiting their applicability to systems with well-defined crystal structures. Furthermore, GNNs still face limitations in integrating non-numerical properties, such as compound names, into the training process without additional preprocessing steps. These challenges underscore the need for approaches that can seamlessly handle both numerical and non-numerical features in material property predictions while minimizing complex preprocessing requirements.

Language models offer unique advantages by directly utilizing human-readable text data, eliminating the need for elaborate pre-processing while preserving critical information embedded in material descriptions.[13, 14, 15, 16] This streamlines the prediction process. Recent advances in natural language processing, particularly with large language models (LLMs), have introduced transformative possibilities for materials science. This simplifies the prediction process compared to conventional ML approaches, which typically require precise atomic coordinates or extensive preprocessing to generate numerical features. In contrast, LLMs can directly process text-based descriptions. Leveraging this capability, we predict band gap values directly from text-formatted input, bypassing the need for detailed structural data and eliminating complex feature engineering.

Recent advances in natural language processing, particularly with LLMs, have introduced transformative opportunities for materials science. For instance, AlloyBERT demonstrates the potential of transformer-based models to predict material properties from descriptive text.[17] Similarly, AMGPT showcases the benefits of using composition-based input strings and finetuned LLMs, enabling accurate and efficient predictions for materials science tasks.[18, 19] Additionally, CatBERTa, a RoBERTa-based predictive model, has been developed to predict adsorption energy in catalyst systems.[13, 14]

In this study, we explore the use of transformer-based language models, RoBERTa, T5, Llama-3, and MatSciBERT to predict the band gaps of semiconductor materials directly from textual descriptions. These models enable the direct encoding of structured or natural language representations of materials, such as chemical composition, crystal symmetry, and electronic features, without requiring conventional feature engineering. While pretrained language models possess strong linguistic priors, we emphasize that fine-tuning on domain-specific objectives is essential for adapting them to materials property prediction. We implement task-specific regression heads on top of each model and finetune them on a curated dataset of inorganic compounds. This approach allows models to learn mappings from text-based input to scalar band gap values. These models provide a flexible framework for property prediction from textual materials data, extending the application of language models beyond conventional natural language tasks into scientific domains such as materials informatics.

Methods

RoBERTa

RoBERTa (Robustly Optimized BERT Approach) is an encoder-only transformer that refines BERT’s pretraining strategy through dynamic masking and longer sequence training. [20] In this work, we employ the RoBERTa model (12 layers, 12 attention heads, 768 hidden units) as one of the backbone architectures for band gap prediction. Its bidirectional attention enables effective contextual encoding of structured text inputs derived from material features. Rather than emphasizing linguistic tasks, our framework fine-tunes RoBERTa to learn correlations between textual feature descriptions and scalar band gap values. This objective-specific adaptation allows the model to repurpose general language representations for scientific property prediction.

T5

T5 (Text-to-Text Transfer Transformer) is a unified encoder–decoder architecture that reformulates all NLP tasks as text-to-text transformations. [21] We employ the T5-small (6 encoder and 6 decoder layers, 512 hidden size, 8 attention heads) to explore how a sequence-to-sequence framework can be applied to scientific regression problems. Throughout this manuscript, we refer to this model simply as T5.

Pretrained with a span corruption objective, T5 learns to reconstruct masked text segments, promoting stronger global reasoning than token-level masking. In our framework, we utilize only the encoder output and append a custom regression head for band gap prediction. This setup leverages T5’s contextual encoding ability while avoiding unnecessary generative complexity, allowing efficient adaptation to structured material descriptions and scalar property prediction.

Llama-3

Llama-3 is a family of decoder-only transformer models developed by Meta, designed to provide state-of-the-art performance with efficient scaling across parameter sizes. [22, 23] In this work, we specifically use the lightweight Llama-3.2-1B variant (approximately one billion parameters, embedding size 2048), which offers a strong balance between performance and computational efficiency. Throughout this manuscript, we refer to this model simply as Llama-3.

We adapt Llama-3 by attaching a custom regression head to the decoder output for band gap prediction. Architectural components such as RMSNorm, SwiGLU activation, and Rotary Positional Embeddings improve training stability and contextual reasoning, while the SentencePiece-based tokenizer effectively encodes scientific symbols and numerical expressions. This configuration enables Llama-3 to extend its pretrained linguistic representations toward modeling structured material descriptions for quantitative property prediction.

MatSciBERT

MatSciBERT is a BERT-based language model pretrained on materials science literature to capture domain-specific terminology, chemical formulas, and structural descriptions. [24] It uses the standard BERT-base architecture with 12 encoder layers, 768-dimensional embeddings, 12 attention heads, and a 3072-dimensional feed-forward layer, while its masked language modeling pretraining is optimized for the symbolic and long-tail vocabulary common in scientific texts. This provides stronger representations for materials-related language than general-purpose models.

In our framework, we employ the MatSciBERT encoder with a custom regression head to predict band gaps from structured material descriptions. This design leverages MatSciBERT’s domain-aware embeddings while maintaining architectural simplicity, allowing efficient adaptation to property prediction tasks within materials informatics.

Shallow ML Models

We implemented three conventional regression algorithms, Random Forest (RF), Support Vector Regression (SVR), and Extreme Gradient Boosting (XGBoost), using the same dataset of structured material descriptors. All models were trained to predict the band gap values from numerical features extracted from the materials database.

The RF model was implemented using scikit-learn’s RandomForestRegressor. The model constructs an ensemble of decision trees trained on randomly sampled subsets of both data and features, and outputs the average prediction across all trees. A five-fold cross-validated grid search was used to optimize key hyperparameters. The final configuration employed 1,000 trees, a maximum depth of 50, a minimum of two samples per split, and two samples per leaf.

For the SVR, we used scikit-learn’s SVR implementation with a radial basis function (RBF) kernel. The model projects the input features into a high-dimensional space where a linear regression is performed within an ϵ\epsilon-insensitive margin. Grid search with five-fold cross-validation determined the optimal hyperparameters: a penalty parameter C=5000C = 5000, ϵ=0.1\epsilon= 0.1, and an RBF kernel. The high value of CC enables the model to fit complex nonlinear patterns, while ϵ\epsilon controls the tolerance for prediction deviations near the regression boundary.

The XGBoost model was implemented using the XGBRegressor from the xgboost library. XGBoost builds an ensemble of boosted decision trees, where each successive tree is trained to correct the residual errors of the previous ensemble. Grid search with five-fold cross-validation identified the best configuration: 2,000 trees, maximum depth of 9, learning rate of 0.1, subsample ratio of 1.0, and column sampling ratio of 0.6. These settings yield deep, expressive trees while maintaining regularization through feature subsampling. The model objective was set to minimize the squared error loss.

Dataset

In this study, we utilized the AFLOW database, a comprehensive open repository for computational materials science that contains extensive information on inorganic crystalline materials and their properties. [25, 26] Band gap calculations in AFLOW combine first-principles methods with empirical corrections through an automated workflow. The framework uses VASP to perform DFT calculations with the GGA-PBE functional for standard compounds while applying the GGA+U method for strongly correlated systems containing d- and f-block elements. To address GGA’s tendency to underestimate band gaps, AFLOW employs an empirical correction scheme based on a linear regression model derived from 102 benchmark compounds with known experimental values. [25, 27, 28] This systematic approach, along with the database’s vast size and rich feature space, makes AFLOW particularly well-suited for ML tasks aimed at material property prediction.

For our specific analysis, we selected a subset of 27,600 materials with band gap values ranging between 0 and 5 eV (inclusive). This range was chosen because it encompasses the most relevant band gap values for semiconductors, which are of particular interest in materials science and electronic applications. The lower bound of 0 eV represents materials with metallic behavior, where there is no electronic band gap. Materials that have band gap higher than 5 eV are insulating materials, which have too large band gaps and do not conduct electricity under normal conditions. [3, 29] By focusing on materials within this range of 0-5 eV band gaps, we ensure that our model targets materials with practical applications in electronics and optoelectronics.

The dataset was divided into training, validation, and test sets to ensure reliable evaluation and optimization of the model. Specifically, 10% of the data was reserved for the test set to evaluate the final performance of the model. The remaining 90% was further split into 80% for training and 20% for validation, ensuring sufficient data for model training while retaining a representative validation set. This splitting strategy ensured the distribution of band gap values across all subsets, minimizing sampling bias and enhancing representativeness.

Text Data Format

To investigate the impact of input data representation on model performance, we employed two formats for encoding material property information as text. The first format consists of structured strings, where material attributes, such as chemical composition, crystal structure, and electronic features, were compiled into a consistent, template-based layout. This format emphasizes uniformity and feature alignment across samples, providing a well-controlled structure for the language models to process.

The second format consists of natural language descriptions generated using OpenAI’s GPT-3.5 Turbo API. The same core features were provided to the API to produce narrative-style descriptions, introducing greater linguistic variability and a more conversational tone. Prompts were configured to ensure descriptions remained within a 512-token limit to maintain compatibility with the tokenizer constraints of RoBERTa, and to accommodate input length limits for T5 and Llama-3 as well.

Both formats were applied uniformly across all three models: RoBERTa, T5, Llama-3, and MatSciBERT. The models processed these inputs through their native tokenization pipelines, without additional handcrafted feature engineering.

Input Features

We carefully selected features that capture both the chemical composition and structural properties of the materials, ensuring a comprehensive understanding of their electronic characteristics, especially the band gap. The selected features include chemical formula, atomic species, valence electron count, crystal symmetry, and magnetic properties, all of which are known to play critical roles in determining the electronic structure and band gap of materials. A complete list of the 23 selected features, categorized by their respective domains, is provided in Table 1.

FeatureDescription
CompoundChemical formula of the material, representing its chemical composition
SpeciesList of atomic species constituting the material
CompositionProportion of each element in the material
Valence cell (iupac)Total number of valence electrons in the unit cell, calculated according to IUPAC standards
Species pseudopotentialType of atomic pseudopotentials used for calculations
Crystal classDescribing the symmetry properties of the crystal
Crystal familyIndicating the basic geometric features of the crystal
Crystal systemDescribing the shape and symmetry of the unit cell
Fractional coordinatesRepresenting the relative positions of atoms in the unit cell
Lattice parametersThe edge lengths and angles of the unit cell
Lattice systemDescribing the basic geometric features of the unit cell
Lattice variationProviding a more detailed description of the lattice
Space group of the structureDescribing the symmetry of the crystal
Space group change looseSpace group determined under looser conditions for crystal structure relaxation, potentially leading to larger symmetry changes
Space group change tightSpace group determined under stricter conditions for structure relaxation, resulting in fewer symmetry changes
Point group orbifoldDescribing the topological properties of the point group
Point group orderIndicating the number of symmetry operations in the point group
Point group structureDescribing the geometric features of the point group
Point group typeClassifying the symmetry properties of the point group
Magnitude of magnetic moment for each atomDescribing the local magnetism of the material
Magnetization intensity of each atomRepresenting magnetism at the atomic scale
Total magnetization intensity of the entire unit cellDescribing the overall magnetism of the material
DensityDensity of the material

Table 1. Selected feature list. Each feature is accompanied by a specific description explaining its physical significance and contribution to material characterization.

The chemical formula represents the basic building blocks of the material, providing critical information about its stoichiometry and composition. The nature and type of atoms constituting the material greatly influence its electronic properties.[9, 30, 31] Specifically, the elemental types and their ratios determine atomic energy levels, such as ss, pp, and dd orbitals, as well as bond types like covalent and ionic bonds, and electronegativity differences. These factors affect the relative positions and separation of the valence and conduction bands. In ionic compounds, a larger electronegativity difference leads to a greater energy separation between the valence band, which is primarily formed by anion orbitals, and the conduction band, which is mainly formed by cation orbitals. This typically results in a larger band gap. For example, II-VI compounds often have larger band gaps compared to III-V compounds.[32]

For example, the number of valence electrons of each species is crucial for band gap predictions.[33] The total number of valence electrons per unit cell governs how the electronic bands are filled and directly influences the position of the Fermi level.[34] Differences in valence electron count and orbital configuration among elements or compounds can lead to significant variations in both the magnitude and nature of the band gap. Materials with stronger orbital overlap and higher structural symmetry tend to have wider bands and smaller, often direct, band gaps, whereas weaker overlap or lower symmetry can result in indirect or larger gaps, ultimately determining the optical and transport properties of the material.[35]

Additionally, we paid particular attention to structural features, including crystal class, family, and system, as well as lattice parameters, which include the dimensions and angles of the unit cell. These factors not only shape the arrangement of atoms and their interactions but also define the symmetry and geometric properties of the crystal, directly influencing the distribution of electronic states within the energy bands.[28, 36, 37] The crystal’s symmetry determines the shape of the Brillouin zone and band degeneracy. High symmetry, as seen in cubic systems, often leads to high band degeneracy at high-symmetry points, where multiple electrons have the same energy, resulting in a relatively simple band structure. In contrast,

lower symmetry, as in orthorhombic systems, can cause band splitting, which may change the band gap’s size and type.[38] The space group and point group information were included to account for the effects of symmetry on electronic states and band splitting. Magnetic properties, such as atom magnetic moments and cell magnetization, were also considered due to their relationship with spin distribution and electronegativity.[3] We specifically chose properties derived from relaxed structures in which the structural configurations have been optimized to minimize energy and stress, ensuring that the atoms are in their equilibrium positions.

Results and Discussion

Framework

We developed a language model-based framework to predict the band gaps of semiconductor materials using transformer-based language models: RoBERTa, T5, Llama-3, and MatSciBERT. As shown in Figure 1a, we constructed the dataset from the AFLOW database by extracting relevant material features and transforming them into textual formats suitable for language model inputs. Two types of textual representations were used (Figure 1b): a structured string format that followed a fixed template, and a more flexible natural language description generated using the GPT-3.5 Turbo API. These formats enabled us to assess how each model handles both highly regular and semantically rich input styles. Additional examples of both the structured strings and natural language descriptions are provided in the Supporting Information.

Overview of the proposed band gap prediction framework

Figure 1. Overview of the proposed band gap prediction framework. a The pipeline starts from the AFLOW dataset, followed by feature selection, dataset preparation, and LLM model training for final band gap prediction. b Two input formats are illustrated. string-based representation using direct feature values and description-based format generated by GPT-3.5 turbo. c Visualization of the finetuning process. The input text undergoes tokenization and embedding through multiple model architectures (RoBERTa, T5, Llama-3, MatSciBERT), followed by a custom regression head for prediction. d Demonstrates the Transformer encoder and the multi-head attention mechanism with Query (QQ), Key (KK), and Value (VV) operations.

At the core of our approach are transformer-based language models, each differing in architecture, tokenization strategy, and training strategy. Our approach is built on transformer-based language models with distinct architectures and training schemes. RoBERTa and MatSciBERT share an encoder-only design, but MatSciBERT is additionally pretrained on materials science texts for domain-specific understanding. T5 uses an encoder–decoder structure, while Llama-3 is a decoder-only model. Each model tokenizes the input text using its native tokenizer, such as byte-level BPE or SentencePiece, and processes the sequence to generate contextual embeddings (Figure 1c). These embeddings are passed through a custom regression head to produce a scalar band gap prediction.

Model Performance

We evaluated the performance of four transformer-based language models—RoBERTa, T5, Llama-3, and MatSciBERT—on the task of predicting semiconductor band gaps from text-based material descriptions. We also included shallow machine learning baselines using the same input format, including SVR, random forest, and XGBoost. GNNs were not considered, as their graph-based input representations do not align with the feature-based approach used here. Model accuracy was quantified using three metrics: mean absolute error (MAE), root mean square error (RMSE), and the coefficient of determination (R2R^2). The results are summarized in Table 2, with parity plots shown in Figure 2.

ModelModel TypeMAE (eV)RMSE (eV)R2R^2
SVRShallow ML0.601±0.0100.601 \pm0.0100.844±0.0080.844 \pm0.0080.600±0.0080.600 \pm0.008
Random ForestShallow ML0.385±0.0060.385 \pm0.0060.609±0.0060.609 \pm0.0060.792±0.0050.792 \pm0.005
XGBoostShallow ML0.318±0.0050.318 \pm0.0050.537±0.0050.537 \pm0.0050.838±0.0040.838 \pm0.004
RoBERTa(string)_{\text{(string)}}LLM (Encoder)0.325±0.0060.325 \pm0.0060.447±0.0050.447 \pm0.0050.855±0.0040.855 \pm0.004
RoBERTa(description)_{\text{(description)}}LLM (Encoder)0.421±0.0070.421 \pm0.0070.590±0.0060.590 \pm0.0060.797±0.0060.797 \pm0.006
T5(string)_{\text{(string)}}LLM (Encoder-Decoder)0.301±0.0070.301 \pm0.0070.448±0.0060.448 \pm0.0060.861±0.0050.861 \pm0.005
T5(description)_{\text{(description)}}LLM (Encoder-Decoder)0.446±0.0110.446 \pm0.0110.615±0.0080.615 \pm0.0080.759±0.0080.759 \pm0.008
Llama-3(string)_{\text{(string)}}LLM (Decoder)0.248±0.0060.248 \pm0.0060.345±0.0050.345 \pm0.0050.891±0.0040.891 \pm0.004
Llama-3(description)_{\text{(description)}}LLM (Decoder)0.335±0.0080.335 \pm0.0080.473±0.0060.473 \pm0.0060.843±0.0540.843 \pm0.054
MatSciBERT(string)_{\text{(string)}}LLM (Encoder)0.288±0.0070.288 \pm0.0070.407±0.0060.407 \pm0.0060.871±0.0040.871 \pm0.004
MatSciBERT(description)_{\text{(description)}}LLM (Encoder)0.366±0.0090.366 \pm0.0090.503±0.0070.503 \pm0.0070.808±0.0070.808 \pm0.007

Table 2. Comparison of model performance across different ML approaches. For LLMs, results are shown as structured string / natural language description. Best performance per metric is shown in bold, and second-best performance is underlined.

Parity plots for band gap predictions across models

Figure 2. Parity plots for band gap predictions across models: a SVR, b XGBoost, c Random Forest, d RoBERTa, e T5, f Llama-3, g MatSciBERT

Across all models, inputs in structured string format consistently outperformed descriptive natural language inputs, as shown in Table 2. Both input types are derived from the same underlying features, so the richness of information is equivalent; the difference arises from how the information is presented. Structured strings provide a consistent, systematic representation that reduces variability, making it easier for the model to learn direct mappings between features and target properties. In contrast, language descriptions introduce variability in phrasing, terminology, and length, requiring the model to first infer the relevant features from context. This adds complexity and can obscure patterns, even though the same information is present.

Among the transformer-based models, Llama-3 with structured input achieved the best overall performance, with a mean absolute error of 0.248 eV and R2R^2 of 0.891 (Table 2).

Following closely, MatSciBERT delivered a strong performance with an MAE of 0.288 eV and an R2R^2 of 0.872. While RoBERTa and T5 also outperformed the shallow ML models, their performance was lower when using GPT-generated descriptions compared to structured string inputs.

These results highlight several insights. First, language models can predict band gaps directly from text-based inputs without any feature preprocessing. Given its large number of parameters, it is not surprising that Llama-3 performed the best. More interestingly, when comparing RoBERTa and MatSciBERT, which share similar architectures, MatSciBERT achieved better results despite having fewer parameters. This suggests that the training strategy and the use of a domain-specific corpus are critical. Pretraining on materials science literature enables the model to more effectively capture the relationship between material features and band gap values.

Even though we evaluated multiple types of language models, it is difficult to generalize performance purely based on architecture since the number of parameters varies greatly. Nonetheless, it is important to demonstrate that text-based band gap prediction works across different model types. It is also not surprising that language model approaches outperform shallow machine learning methods. To clarify, both shallow ML models and language models use the same feature set, so the richness of information is identical. The superior performance of language models indicates their enhanced capability to learn complex correlations between the provided features and the band gap.

Layer Freezing Analysis

To investigate how pretrained representations in each model contribute to band gap prediction, we conducted layer-freezing experiments on RoBERTa, T5, Llama-3, and MatSciBERT using structured string inputs. In these experiments, we progressively froze layers of the transformer backbone and fine-tuned only a portion of the model, while training a custom regression head on top of the frozen or partially frozen representations. The goal was not to reduce computational cost, but to probe which layers carry the most task-relevant information. Table 3 reports the MAE, RMSE, and R2R^2 values for each model under various freezing setups. This analysis provides insight into how much domain-specific fine-tuning is necessary and which parts of the pretrained model are most informative for the downstream task, highlighting the layers that contribute most strongly to predictive accuracy.

ModelFreezing StrategyParameters (count, %)MAE (eV)RMSE (eV)R2R^2
RoBERTaFully finetuned (no freezing)125,236,993 (100%)0.325 ± 0.0060.447 ± 0.0050.855 ± 0.004
RoBERTaFreeze first layer118,149,121 (94.3%)0.328 ± 0.0080.448 ± 0.0060.848 ± 0.005
RoBERTaFreeze all but final 3 layers61,446,145 (49.1%)0.388 ± 0.0090.510 ± 0.0070.817 ± 0.006
RoBERTaFreeze all but final layer8,269,825 (6.6%)0.509 ± 0.0120.648 ± 0.0090.721 ± 0.009
RoBERTaFreeze all layers591,361 (0.5%)0.663 ± 0.0160.826 ± 0.0110.563 ± 0.013
T5Fully finetuned (no freezing)60,769,793 (100%)0.301 ± 0.0070.448 ± 0.0060.861 ± 0.005
T5Freeze first layer57,622,785 (94.8%)0.350 ± 0.0080.504 ± 0.0070.849 ± 0.006
T5Freeze all but final 3 layers21,830,401 (19.8%)0.367 ± 0.0090.516 ± 0.0070.832 ± 0.006
T5Freeze all but final layer7,671,553 (7.0%)0.598 ± 0.0140.784 ± 0.0110.619 ± 0.011
T5Freeze all layers592,129 (0.5%)0.792 ± 0.0190.981 ± 0.0130.420 ± 0.014
Llama-3Fully finetuned (no freezing)1,237,915,649 (100%)0.248 ± 0.0060.345 ± 0.0050.891 ± 0.004
Llama-3Freeze first layer1,177,094,145 (95.1%)0.279 ± 0.0070.426 ± 0.0060.878 ± 0.004
Llama-3Freeze all but final 3 layers447,236,097 (36.1%)0.318 ± 0.0080.474 ± 0.0060.851 ± 0.005
Llama-3Freeze all but final layer325,593,089 (26.3%)0.424 ± 0.0100.576 ± 0.0080.793 ± 0.007
Llama-3Freeze all layers2,101,249 (0.2%)0.716 ± 0.0170.893 ± 0.0120.518 ± 0.013
MatSciBERTFully finetuned (no freezing)110,509,825 (100%)0.288 ± 0.0070.407 ± 0.0060.871 ± 0.004
MatSciBERTFreeze first layer103,421,953 (93.6%)0.293 ± 0.0070.404 ± 0.0050.874 ± 0.004
MatSciBERTFreeze all but final 3 layers22,445,569 (20.3%)0.340 ± 0.0080.454 ± 0.0060.849 ± 0.005
MatSciBERTFreeze all but final layer8,269,825 (7.5%)0.415 ± 0.0100.539 ± 0.0070.798 ± 0.007
MatSciBERTFreeze all layers1,181,953 (1.1%)0.827 ± 0.0200.994 ± 0.0130.385 ± 0.015

Table 3. Comparison of layer freezing strategies across RoBERTa, T5, Llama-3, and MatSciBERT using structured string inputs. The fully finetuned (non-frozen) results are included for reference. Parameter counts and percentages represent the number of trainable parameters relative to the no-freezing case. All values are reported as mean ± standard deviation.

Throughout this study, we refer to the original language models, trained on general text corpora prior to any materials-specific adaptation, as “pretrained” models. After supervised training on the band gap prediction task, we refer to the models as “finetuned.”

In all four models, predictive performance improved progressively as more transformer layers were unfrozen, highlighting the importance of task-specific fine-tuning in scientific regression tasks, as shown in Table 3. While LLMs capture broadly useful representations during pretraining, these representations alone are insufficient to achieve optimal accuracy in domain-specific prediction tasks without additional adaptation. As more layers are involved in fine-tuning, the models can better capture feature–property correlations. Interestingly,

MatSciBERT exhibits slightly better RMSE and R2R^2 scores compared to its fully finetuned case, which may be attributed to the well-formed representations learned from prior exposure to the materials science domain.

The rough scaling relationship between the number of parameters and prediction accuracy is illustrated in Figure 3. Generally, as the number of trainable parameters increases, MAE decreases, indicating improved predictive performance across models and layer-freezing strategies. Within each model, this relationship is relatively linear in the layer-wise freezing experiments, showing that gradually unfreezing layers consistently improves performance. However, when comparing across different models, the relationship is not strictly linear. This reflects the fact that predictive performance depends not only on the number of parameters but also on other factors, including model architecture, pretraining corpus, task-specific fine-tuning strategies, and the alignment between learned representations and the structured input features. These results emphasize that while pretrained LLMs provide a strong foundation, achieving optimal performance in materials science requires careful selection of trainable layers and fine-tuning strategies, especially under computational or resource constraints.

Trained Parameter Count vs MAE

Figure 3. Scaling behavior of finetuning strategies across transformer-based models. MAE is shown as a function of the number of trainable parameters. Colors indicate different model architectures: RoBERTa (yellow), T5 (green), Llama-3 (blue), and MatSciBERT (red). Marker shapes represent different freezing strategies, from fully finetuned (no freezing) to full layer freezing.

Feature-wise Self-Attention Score

To identify which material features are most emphasized during band gap prediction, we conducted a feature-wise self-attention analysis on Llama-3 and MatSciBERT, the two best-performing models. In transformer architectures, the attention mechanism quantifies how strongly each token attends to others when forming contextual representations, as shown in Figure 1d.

For each model, input text was tokenized (maximum length of 512 tokens, truncation enabled), and attention weights were extracted from the first and last transformer layers. The attention score here refers to the attention weight between the first token, used as the regression header, and all other tokens in the sequence. For each head, the attention scores from the regression token to the rest of the tokens were averaged, and the resulting values were then averaged across all heads within the layer.

Attention scores between the regression header token and all feature tokens were extracted from each transformer layer. For a given layer ll and sample ii, the attention weights were first averaged across all attention heads:

si,l(t)=1H∑h=1Hsi,l,h(t)(1)s_{i,l}(t) = \frac{1}{H}\sum_{h=1}^{H}s_{i,l,h}(t) \tag*{(1)}

where HH is the number of attention heads and si,l,h(t)s_{i,l,h}(t) denotes the attention score from the header token to token tt in head hh.

For each feature ff, we identified its corresponding token span and recorded the maximum attention score within that span:

scorei,l,fraw=max⁡t∈[tstart,tend]si,l(t)(2)\mathrm{score}^{\mathrm{raw}}_{i,l,f} = \max_{t\in[t_{\mathrm{start}},t_{\mathrm{end}}]} s_{i,l}(t) \tag*{(2)}

To ensure comparability across samples, raw scores were normalized to the range [0,1][0, 1] using min–max scaling:

scorei,l,fnorm=scorei,l,fraw−min⁡kscorei,l,krawmax⁡kscorei,l,kraw−min⁡kscorei,l,kraw(3)\mathrm{score}^{\mathrm{norm}}_{i,l,f} = \frac{\mathrm{score}^{\mathrm{raw}}_{i,l,f} - \min_k \mathrm{score}^{\mathrm{raw}}_{i,l,k}}{\max_k \mathrm{score}^{\mathrm{raw}}_{i,l,k} - \min_k \mathrm{score}^{\mathrm{raw}}_{i,l,k}} \tag*{(3)}

where kk indexes all features in sample ii.

Finally, layer-wise feature attention was obtained by averaging normalized scores across all test samples:

Avg_Attentionl,f=1N∑i=1Nscorei,l,fnorm(4)\mathrm{Avg\_Attention}_{l,f} = \frac{1}{N}\sum_{i=1}^{N}\mathrm{score}^{\mathrm{norm}}_{i,l,f} \tag*{(4)}

This procedure provides a consistent measure of how each layer attends to specific material features based on attention scores. It is important to note that these scores do not incorporate the value matrix; therefore, as illustrated in Figure 1d, they do not directly quantify feature importance. Instead, they offer a descriptive view of the model’s focus on the input tokens.

The results reveal clear differences in attention patterns between the first and last layers for both models. In our analysis, the embedding of the first token from the last layer is ultimately used for regression, reflecting how the model allocates attention to optimize prediction performance. For LLaMA-3, attention to geometric features, such as lattice variation, lattice system, and overall geometry, decreases in the final layer, as seen by comparing Figure 4a and b. Although these features are important for structure–property relationships, this pattern suggests that LLaMA-3 struggles to directly extract geometric information from text-based descriptions, instead emphasizing compositional and spin-related features.

Feature-wise self-attention scores for LLaMA-3 and MatSciBERT

Figure 4. Feature-wise self-attention scores for LLaMA-3 and MatSciBERT. a LLaMA-3, first layer; b LLaMA-3, final layer (layer 16); c MatSciBERT, first layer; d MatSciBERT, final layer (layer 12).

MatSciBERT, which benefits from domain-specific pretraining on materials science literature, shows earlier specialization in crystallographic features, as illustrated in Figure 4c.

Similar to LLaMA-3, attention to geometric features declines in later layers, highlighting the difficulty of encoding geometric information purely from text (Figure 4d). However, MatSciBERT distributes attention more broadly in the first layer, with structural symmetry attributes such as point group receiving relatively high scores. By the final layer, attention becomes more selective, focusing on key features such as composition, including species and compound information.

Embedding Map

We conducted embedding space analysis using t-SNE visualizations to investigate how each model organizes material representations before and after fine-tuning. As shown in Figure 5 and 6, we extracted the first-token embeddings from the pretrained and finetuned versions of RoBERTa, T5, Llama-3, and MatSciBERT using structured string inputs, and colored the points by crystal system. In the “pretrained” condition, which corresponds to the original language model weights before finetuning on the band gap prediction task, the embeddings reflect structural signals learned from general language corpora. The “fine-tuned” condition represents embeddings generated after the models were fully fine-tuned for band gap prediction.

t-SNE visualizations of embeddings colored by crystal system

Figure 5. t-SNE visualizations of embeddings colored by crystal system. a-d show results from the pretrained models: a RoBERTa, b T5, c Llama-3, d MatSciBERT. e-h show results from the corresponding finetuned models: e RoBERTa, f T5, g Llama-3, h MatSciBERT.

In the pretrained state, embeddings from all four models exhibit meaningful clustering according to crystal system, as shown in Figure 5a-Figure 5d. In contrast, when coloring by bandgap (Figure 6a-Figure 6d), no clear clustering is observed. This structure-aware behavior likely arises from explicit input features such as crystal class, space group, and lattice parameters, which correlate with symmetry. While these features can separate materials by structural family, they do not provide sufficient signal to predict electronic properties like bandgap. In other words, without finetuning, the models lack task-specific supervision to map structural cues to quantitative property outcomes.

t-SNE visualizations of embeddings for bandgap prediction

Figure 6. t-SNE visualizations of embeddings for bandgap prediction. a-d show results from the pretrained models: a RoBERTa, b T5, c Llama-3, d MatSciBERT. e-h show results from the corresponding finetuned models: e RoBERTa, f T5, g Llama-3, h MatSciBERT.

After finetuning on the bandgap prediction task, however, the clustering behavior changes markedly: embeddings now group primarily by bandgap rather than by crystal system, as visualized in Figure 5e-Figure 5h and Figure 6e-Figure 6h. In particular, Llama-3 shows a significant loss of crystal system clustering while forming a near-linear embedding map based on bandgap, which likely contributes to its superior predictive performance. The result is an embedding space where materials with similar bandgaps are positioned closer together, independent of their crystallographic classification. This transformation illustrates how supervised training can reorient general-purpose language representations toward property-specific scientific objectives.

T5 produces a qualitatively different t-SNE embedding: unlike the roughly circular maps seen in most other models, T5’s embedding forms a curvy, elongated shape. We note that the models have differing embedding dimensions, and t-SNE can distort distances depending on dimensionality and distribution. Therefore, some apparent differences may reflect visualization artifacts or latent space geometry rather than intrinsic model behavior. Further quantitative analysis of cluster separation and latent structure would be needed to confirm these patterns.

Conclusion

In this study, we investigated the use of transformer-based language models, RoBERTa, T5, Llama-3, and MatSciBERT for predicting the band gaps of semiconductor materials directly from textual inputs. We demonstrated that these models can learn meaningful feature–property relationships without relying on complex feature preprocessing or structure graph of atomic coordinates. Llama-3 achieving the highest accuracy using structured inputs (MAE 0.248 eV, R2R^2 0.891). Even with natural language descriptions, the models captured relevant patterns, highlighting their flexibility for scenarios lacking structured data.

Overall, this work highlights the potential of both general-purpose and domain-specific language models as flexible, scalable, and efficient tools for materials property prediction. By enabling direct prediction of properties from human-readable text, these models remove the need for extensive feature engineering or graph-based structure encoding, allowing rapid, end-to-end property estimation from structured or natural language material descriptions. The results also demonstrate that pretrained language models can provide interpretable insights through attention and embedding analyses, identifying the material features most relevant to the target property. .

Code Availability Statement

The Python code in this study is available on GitHub at the following link: https://github.com/yingtiny/bandgap_prediction_RoBERTa.

Technology Use Disclosure

We used ChatGPT and Claude to help with grammar and typographical corrections during the preparation of this manuscript. The authors have carefully reviewed, verified, and approved all content to ensure accuracy and integrity.

Supporting Information: Text to Band Gap: Pre-trained Language Models as Encoders for Semiconductor Band Gap Prediction

Ying-Ting Yeh; Janghoon Ock; Achuth Chandrasekhar; Shagun Maheshwari; Amir Barati Farimani

String and Description Examples

CrO3_3Ta

String:

compound: Cr1O3Ta1, species: [’Cr’, ’O’, ’Ta’], composition: [1, 3, 1], density: 7.274, valence_cell_iupac: 17, species_pp: [’Cr_pv’, ’O’, ’Ta_pv’], spinD: [4.555, 0.016, 0.021, 0.008, 0.061], spin_atom: 1.0, spin_cell: 5.001, crystal_class: hexoctahedral, crystal_family: cubic, crystal_system: cubic, positions_fractional: [[0, 0, 0], [0, 0.5, 0.5], [0.5, 0, 0.5], [0.5, 0.5, 0], [0.5, 0.5, 0.5]], geometry: [4.003, 4.003, 4.003, 90, 90, 90], lattice_system_relax: cubic, lattice_variation_relax: CUB, spacegroup_relax: 221, sg: [’Pm-3m #221’, ’Pm-3m #221’, ’Pm-3m #221’], sg2: [’Pm-3m #221’, ’Pm-3m #221’, ’Pm-3m #221’], point_group_orbifold: *432, point_group_order: 48, point_group_structure: 2 x symmetric, point_group_type: centrosymmetric

Description:

The compound CrO3Ta\mathrm{CrO_3Ta} features a composition of 1 Cr atom, 3 O atoms, and 1 Ta atom, with a density of 7.27364 g/cm3^3. It belongs to the cubic crystal system and has a hexoctahedral crystal class. The lattice system is relaxed cubic with lattice parameters a=b=c=4.00287a = b = c = 4.00287 Å and angles α=β=γ=90∘\alpha= \beta= \gamma= 90^\circ.

The compound has a valence of 17 according to the IUPAC system and crystallizes in the space group Pm-3m #221. It exhibits spin values of 1.00013 at the atomic level and 5.00063 at the cell level. The structure is highly symmetric, belonging to the *432 point group with an order of 48 and a centrosymmetric configuration.

The atoms are located at fractional coordinates: (0, 0, 0), (0, 0.5, 0.5), (0.5, 0, 0.5), (0.5, 0.5, 0), and (0.5, 0.5, 0.5). The atomic species present are Cr, O, and Ta, with the pseudopotential designations Cr_pv, O, and Ta_pv, respectively.

Bi1Dy1Ni1\mathrm{Bi}_1\mathrm{Dy}_1\mathrm{Ni}_1

String:

compound: Bi1Dy1Ni1, species: [’Bi’, ’Dy’, ’Ni’], composition: [1, 1, 1], density: 10.599, valence_cell_iupac: 13, species-pp: [’Bi_d’, ’Dy_3’, ’Ni_pv’], spinD: [0, 0, 0], spin_atom: 0.0, spin_cell: 0.0, crystal_class: tetrahedral, crystal_family: cubic, crystal_system: cubic, positions_fractional: [[0, 0, 0], [0.5, 0.5, 0.5], [0.25, 0.25, 0.25]], geometry: [4.568, 4.568, 4.568, 60, 60, 60], lattice_system_relax: cubic, lattice_variation_relax: FCC, spacegroup_relax: 216, sg: [’F-43m #216’, ’F-43m #216’, ’F-43m #216’], sg2: [’F-43m #216’, ’F-43m #216’, ’F-43m #216’], point_group_orbifold: *332, point_group_order: 24, point_group_structure: symmetric, point_group_type: none

Description:

This material is a cubic compound with the chemical formula BiDyNi. It has a density of 10.5987 g/cm310.5987\ \mathrm{g}/\mathrm{cm}^3 and a valence of 13 according to the IUPAC system. The crystal structure is tetrahedral within the cubic crystal family and system. The lattice system is relaxed cubic with a face-centered cubic (FCC) lattice variation. The space group is F-43m #216, and the point group is *332 with an order of 24, showing a symmetric structure. The atomic positions in the unit cell are at (0,0,0), (0.5,0.5,0.5), and (0.25,0.25,0.25). The atomic species present are Bi, Dy, and Ni, with spins of 0 for each atom. The geometry of the unit cell is characterized by lattice parameters of a=b=c=4.567913a=b=c=4.567913 Å and α=β=γ=60∘\alpha=\beta=\gamma=60^\circ. The species have the configurations Bi_d, Dy_3, and Ni_pv respectively.

Au2Bi2Li4\mathrm{Au}_{2}\mathrm{Bi}_{2}\mathrm{Li}_{4}

String:

compound: Au2Bi2Li4, species: [’Au’, ’Bi’, ’Li’], composition: [2, 2, 4], density: 8.068, valence_cell_iupac: 24, species_pp: [’Au’, ’Bi_d’, ’Li_sv’], spinD: [0, 0, 0, 0, 0, 0, 0, 0], spin_atom: 0.0, spin_cell: 0.0, crystal_class: orthorhombic-bipyramidal, crystal_family: orthorhombic, crystal_system: orthorhombic, positions_fractional: [[0, 0, 0], [0, 0, 0.5], [0.662, 0.662, 0.25], [0.338, 0.338, 0.75], [0.474, 0.12, 0.25], [0.526, 0.88, 0.75], [0.12, 0.474, 0.25], [0.88, 0.526, 0.75]], geometry: [5.563, 5.563, 5.638, 90, 90, 97.937], lattice_system_relax: orthorhombic, lattice_variation_relax: ORCC, spacegroup_relax: 63, sg: [’Cmcm #63’, ’Cmcm #63’, ’Cmcm #63’], sg2: [’Cmcm #63’, ’Cmcm #63’, ’Cmcm #63’], point_group_orbifold: *222, point_group_order: 8, point_group_structure: 2_x_dihedral, point_group_type: centrosymmetric

Description:

The compound Au2Bi2Li4\mathrm{Au}_{2}\mathrm{Bi}_{2}\mathrm{Li}_{4} features a unique orthorhombic crystal structure with a crystal class of orthorhombic-bipyramidal, belonging to the orthorhombic crystal family and system. The material has a density of 8.06805 g/cm3^{3} and a valence cell of 24.

The chemical composition consists of 2 atoms of Au, 2 atoms of Bi, and 4 atoms of Li. The lattice system is orthorhombic, with lattice parameters a=b=5.562936a=b=5.562936 Å, c=5.638398c=5.638398 Å, and angles α=β=90∘\alpha=\beta=90^\circ, γ=97.937∘\gamma=97.937^\circ. The space group is Cmcm #63, with a relaxed lattice system of orthorhombic and lattice variation of ORCC.

The atoms are positioned in the crystal structure at fractional coordinates: (0, 0, 0), (0, 0, 0.5), (0.662, 0.662, 0.25), (0.338, 0.338, 0.75), (0.474, 0.12, 0.25), (0.526, 0.88, 0.75), (0.12, 0.474, 0.25), and (0.88, 0.526, 0.75).

The point group characteristics include a point group orbifold of *222, an order of 8, a structure of 2_x_dihedral, and a centrosymmetric type. Spin properties indicate zero spin for both individual atoms and the overall unit cell.

Ag2_2CrPt

String:

compound: Ag2Cr1Pt1, species: [’Ag’, ’Cr’, ’Pt’], composition: [2, 1, 1], density: 12.174, valence_cell_iupac: 20, species_pp: [’Ag’, ’Cr-pv’, ’Pt’], spinD: [0, 0, 0, 0], spin_atom: 0.0, spin_cell: 0.0, crystal_class: tetrahedral, crystal_family: cubic, crystal_system: cubic, positions_fractional: [[0, 0, 0], [0.25, 0.25, 0.25], [0.5, 0.5, 0.5], [0.75, 0.75, 0.75]], geometry: [4.469, 4.469, 4.469, 60, 60, 60], lattice_system_relax: cubic, lattice_variation_relax: FCC, spacegroup_relax: 216, sg: [’F-43m #216’, ’F-43m #216’, ’F-43m #216’], sg2: [’F-43m #216’, ’F-43m #216’, ’F-43m #216’], point_group_orbifold: *332, point_group_order: 24, point_group_structure: symmetric, point_group_type: none

Description:

The material is a compound with the chemical formula Ag2_2CrPt, consisting of silver (Ag), chromium (Cr), and platinum (Pt) in a ratio of 2:1:1. It has a density of 12.1739 g/cm3^3 and a valence of 20 according to the IUPAC standard. The crystal structure is cubic with a tetrahedral arrangement. The lattice system is cubic with a face-centered cubic (FCC) lattice variation, and the space group is F-43m #216. The point group is *332, indicating a symmetric structure with 24-fold rotational symmetry.

The system exhibits no net magnetic moment, with spin values of 0.0 at both the atom and cell levels. Atomic positions are defined by fractional coordinates: (0, 0, 0), (0.25, 0.25, 0.25), (0.5, 0.5, 0.5), and (0.75, 0.75, 0.75) within the unit cell. The crystal geometry is characterized by lattice parameters a=b=c=4.4694a = b = c = 4.4694 Å and angles α=β=γ=60∘\alpha= \beta= \gamma= 60^\circ.

References

  1. [1]Yu, P. Y.; Cardona, M. Fundamentals of Semiconductors: Physics and Materials Properties, 4th ed.; Springer, 2010.
  2. [2]Kim, S.; Lee, M.; Hong, C.; Yoon, Y.; An, H.; Lee, D.; Jeong, W.; Yoo, D.; Kang, Y.; Youn, Y.; others A band-gap database for semiconducting inorganic materials calculated with hybrid functional. Scientific Data 2020, 7, 387.DOI
  3. [3]Masood, H.; Sirojan, T.; Toe, C. Y.; Kumar, P. V.; Haghshenas, Y.; Sit, P. H.; Amal, R.; Sethu, V.; Teoh, W. Y. Enhancing prediction accuracy of physical band gaps in semiconductor materials. Cell Reports Physical Science 2023, 4.DOI
  4. [4]Koch, W.; Holthausen, M. C. A chemist’s guide to density functional theory; John Wiley & Sons, 2015.
  5. [5]Kohn, W.; Sham, L. J. Self-consistent equations including exchange and correlation effects. Physical review 1965, 140, A1133.DOI
  6. [6]Schleder, G. R.; Padilha, A. C.; Acosta, C. M.; Costa, M.; Fazzio, A. From DFT to machine learning: recent approaches to materials science–a review. Journal of Physics: Materials 2019, 2, 032001.DOI
  7. [7]Wang, T.; Tan, X.; Wei, Y.; Jin, H. Accurate bandgap predictions of solids assisted by machine learning. Materials Today Communications 2021, 29, 102932.
  8. [8]Rajan, A. C.; Mishra, A.; Satsangi, S.; Vaish, R.; Mizuseki, H.; Lee, K.-R.; Singh, A. K. Machine-learning-assisted accurate band gap predictions of functionalized MXene. Chemistry of Materials 2018, 30, 4031–4038.
  9. [9]Zhuo, Y.; Mansouri Tehrani, A.; Brgoch, J. Predicting the band gaps of inorganic solids by machine learning. The journal of physical chemistry letters 2018, 9, 1668–1673.DOI
  10. [10]Faber, F. A.; Hutchison, G. R.; Huang, B.; von Lilienfeld, O. A. Prediction errors of molecular machine learning models lower than hybrid DFT error. Journal of Chemical Theory and Computation 2017, 13, 5255–5264.DOI
  11. [11]Choudhary, K.; DeCost, B. The Atomistic Line Graph Neural Network for improved materials property predictions. npj Computational Materials 2021, 7, 185.DOI
  12. [12]Taniguchi, T.; Hosokawa, M.; Asahi, T. Graph comparison of molecular crystals in band gap prediction using neural networks. ACS omega 2023, 8, 39481–39489.DOI
  13. [13]Ock, J.; Guntuboyina, C.; Barati Farimani, A. Catalyst Energy Prediction with Cat-BERTa: Unveiling Feature Exploration Strategies through Large Language Models. ACS Catalysis 2023, 13, 16032–16044.
  14. [14]Ock, J.; Badrinarayanan, S.; Magar, R.; Antony, A.; Barati Farimani, A. Multimodal language and graph learning of adsorption configuration in catalysis. Nature Machine Intelligence 2024, 1–11.arxiv.org/abs/2401.07408
  15. [15]Guntuboina, C.; Das, A.; Mollaei, P.; Kim, S.; Barati Farimani, A. PeptideBERT: A Language Model Based on Transformers for Peptide Property Prediction. The Journal of Physical Chemistry Letters 2023, 14, 10427–10434, PMID: 37956397.arxiv.org/abs/2309.03099
  16. [16]Pak, P.; Farimani, A. B. AdditiveLLM: Large Language Models Predict Defects in Additive Manufacturing. 2025; https://arxiv.org/abs/2501.17784.
  17. [17]Chaudhari, A.; Guntuboina, C.; Huang, H.; Farimani, A. B. AlloyBERT: Alloy property prediction with large language models. Computational Materials Science 2024, 244, 113256.arxiv.org/abs/2403.19783
  18. [18]Jacobs, R.; Polak, M. P.; Schultz, L. E.; Mahdavi, H.; Honavar, V.; Morgan, D. Regression with Large Language Models for Materials and Molecular Property Prediction. 2024; https://arxiv.org/abs/2409.06080.
  19. [19]Chandrasekhar, A.; Chan, J.; Ogoke, F.; Ajenifuja, O.; Barati Farimani, A. AMGPT: A large language model for contextual querying in additive manufacturing. Additive Manufacturing Letters 2024, 11, 100232.
  20. [20]Liu, Y. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 2019, 364.arxiv.org/abs/1907.11692
  21. [21]Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research 2020, 21, 1–67.arxiv.org/abs/1910.10683
  22. [22]Meta AI LLaMA 3.2 Model Overview. 2024; https://arxiv.org/pdf/2407.21783.
  23. [23]Meta AI LLaMA 3.1 Technical Report. 2023; https://arxiv.org/pdf/2302.13971.
  24. [24]Gupta, T.; Zaki, M.; Krishnan, N. A.; Mausam MatSciBERT: A materials domain language model for text mining and information extraction. npj Computational Materials 2022, 8, 102.arxiv.org/abs/2109.15290
  25. [25]Gossett, E.; Toher, C.; Oses, C.; Isayev, O.; Legrain, F.; Rose, F.; Zurek, E.; Carrete, J.; Mingo, N.; Tropsha, A.; others AFLOW-ML: A RESTful API for machine-learning predictions of materials properties. Computational Materials Science 2018, 152, 134–145.arxiv.org/abs/1711.10744
  26. [26]Taylor, R. H.; Rose, F.; Toher, C.; Levy, O.; Yang, K.; Nardelli, M. B.; Curtarolo, S. A RESTful API for exchanging materials data in the AFLOWLIB. org consortium. Computational materials science 2014, 93, 178–192.
  27. [27]Setyawan, W.; Gaume, R. M.; Lam, S.; Feigelson, R. S.; Curtarolo, S. High-throughput combinatorial database of electronic band structures for inorganic scintillator materials. ACS combinatorial science 2011, 13, 382–390.DOI
  28. [28]Wang, T.; Zhang, K.; Thé, J.; Yu, H. Accurate prediction of band gap of materials using stacking machine learning model. Computational Materials Science 2022, 201, 110899.DOI
  29. [29]Tripathy, S. K.; Pattanaik, A. Optical and electronic properties of some semiconductors from energy gaps. Optical Materials 2016, 53, 123–133.DOI
  30. [30]He, Y.; Cubuk, E. D.; Allendorf, M. D.; Reed, E. J. Metallic metal–organic frameworks predicted by the combination of machine learning methods and ab initio calculations. The journal of physical chemistry letters 2018, 9, 4562–4569.DOI
  31. [31]Khan, A.; Tayara, H.; Chong, K. T. Prediction of organic material band gaps using graph attention network. Computational Materials Science 2023, 220, 112063.DOI
  32. [32]Wei, S.-H.; Zunger, A. Calculated natural band offsets of all II–VI and III–V semiconductors: Chemical trends and the role of cation d orbitals. Applied Physics Letters 1998, 72, 2011–2013.DOI
  33. [33]Huang, Y.; Yu, C.; Chen, W.; Liu, Y.; Li, C.; Niu, C.; Wang, F.; Jia, Y. Band gap and band alignment prediction of nitride-based semiconductors using machine learning. Journal of Materials Chemistry C 2019, 7, 3238–3245.DOI
  34. [34]Hu, C. Modern semiconductor devices for integrated circuits. (No Title) 2010,
  35. [35]Yuan, L.-D.; Deng, H.-X.; Li, S.-S.; Wei, S.-H.; Luo, J.-W. Unified theory of direct or indirect band-gap nature of conventional semiconductors. Physical Review B 2018, 98, 245203.DOI
  36. [36]Zheng, X.; Cohen, A. J.; Mori-Sánchez, P.; Hu, X.; Yang, W. Improving band gap prediction in density functional theory from molecules to solids. Physical review letters 2011, 107, 026403.pubmed.ncbi.nlm.nih.gov/21797627
  37. [37]Na, G. S.; Jang, S.; Lee, Y.-L.; Chang, H. Tuplewise material representation based machine learning for accurate band gap prediction. The Journal of Physical Chemistry A 2020, 124, 10616–10623.DOI
  38. [38]Vasseur, G.; Fagot-Revurat, Y.; Kiernen, B.; Sicot, M.; Malterre, D. Effect of symmetry breaking on electronic band structure: gap opening at the high symmetry points. Symmetry 2013, 5, 344–354.

Paper details

vlnc:W_bg65d6gj
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 4

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents