WorksPhysics and Astronomy
Thermodynamics-inspired Explanations of Artificial Intelligence
Abstract
In recent years, predictive machine learning methods have gained prominence in various scientific domains. However, due to their black-box nature, it is essential to establish trust in these models before accepting them as accurate. One promising strategy for assigning trust involves employing explanation techniques that elucidate the rationale behind a black-box model's predictions in a manner that humans can understand. However, assessing the degree of human interpretability of the rationale generated by such methods is a nontrivial challenge. In this work, we introduce interpretation entropy as a universal solution for assessing the degree of human interpretability associated with any linear model. Using this concept and drawing inspiration from classical thermodynamics, we present Thermodynamics-inspired Explainable Representations of AI and other black-box Paradigms (TERP), a method for generating accurate, and human-interpretable explanations for black-box predictions in a model-agnostic manner. To demonstrate the wide-ranging applicability of TERP, we successfully employ it to explain various black-box model architectures, including deep learning Autoencoders, Recurrent Neural Networks, and Convolutional Neural Networks, across diverse domains such as molecular simulations, text, and image classification.
Introduction
Performing predictions based on observed data is a general problem of interest in a wide range of scientific disciplines. Traditionally, scientists have tackled this problem by developing mathematical models that connect observations with predictions using their knowledge of the underlying physical processes. However, in many practical situations, constructing such explicit models is unfeasible due to a lack of system-specific information.[1] In recent years, an alternative class of purely data-driven approaches involving Artificial Intelligence (AI) has emerged with remarkable success.[2, 3, 4, 5, 6, 7, 8, 9] These methods are often referred to as black-box models, as they don’t rely on a deep understanding of the system’s inner workings and are designed to extract patterns directly from data. However, when it comes to making informed decisions and policies based on these models, this lack of understanding raises concerns.
Recently there has been significant progress in addressing this issue and the proposed approaches can be classified into two categories: (a) AI models that are inherently explainable, or (b) post-hoc explanation schemes for AI models that are not inherently explainable (XAI).[10] Since most of the existing black-box AI are not inherently explainable, the latter class of methods has seen success in generating human comprehensible rationale behind AI predictions.[11] Under this paradigm, different methods have been developed which can be black-box model-specific, or model-agnostic that generate global or locally valid explanations in the form of visual or feature importance attributions.[12, 13, 14] For example, the LIME[15] method constructs a linear surrogate model that locally approximates the behavior of a black-box model. Coefficients associated with each feature of the constructed linear model are then used to attribute feature importance. On the other hand, permutation-based explanation methods such as SHAP[16] are designed to identify non-linear dependence among features at a higher computational cost. Other relevant explanation approaches include gradient-based methods such as layer-wise relevance propagation (LRP),[17] guided back-propagation,[18] integrated gradients[19] etc. Although these methods are designed to rationalize AI predictions, there is a potential issue regarding their level of human interpretability. For instance, when the rationalization involves a high number of correlated features, achieving high human interpretability and consequently establishing trust can be challenging. While quantifying the degree of human interpretability of AI explanation techniques has been an area of active research, the progress has been arguably limited. The progress so far includes methods that construct linear models to approximate AI models, by directly taking the number of model parameters or other similar approaches e.g, Akaike information criterion[20] or Bayesian information criterion[21] as a proxy for human-interpretability.
One of the primary motivations behind our work is the recognition that model complexity can be an insufficient descriptor of human interpretability as shown in Figure 1. In this case, if model complexity is used as a proxy for human interpretability, then both linear models shown in Figure 1(a,Figure 1b) will be assigned the same value as they both have the same number of model parameters. InWe call our approach Thermodynamics-inspired Explainable Representations of AI and other black-box Paradigms (TERP). Owing to its model-agnostic implementation, TERP can be used for explaining predictions from any AI classifier. We demonstrate this generality by explaining the following black-box models in this work: (i) autoencoder based VAMPnets27 for tabular molecular data, (ii) self-attention based vision transformers for images28 and, (iii) attention-based bidirectional long short-term memory (Att-BLSTM) for text29 classification. In particular, the first class of models belongs to an area of research undergoing rapid progress[30, 31] involving molecular dynamics (MD) simulations.[31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43] As researchers with a keen interest in MD simulations, we have observed that the application of AI explanation tools to AI models in this field has been very limited. Consequently, we believe that our proposed method TERP will prove valuable to the broader scientific community focused on this subject.

Figure 1. Model complexity is not a good descriptor for human interpretability. Illustrative linear models (a), & (b) predict a target with the same level of accuracy. Both have the same number of model parameters (six), however, model (b) is significantly more human-interpretable than model (a). In model (b), two out of the six features stand out as most relevant for making predictions while it’s difficult to identify relevant features for model (a).
deed, previous studies[22, 23, 24] have revealed constraints in human cognition arising from a bottleneck in information processing capacity when subjected to different stimuli. Thus, we ground ourselves in the information-theoretic definition of entropy (Sec. II B),[25] and adopt a methodology that views linear model weights as a probability distribution. This allows us to assess differences in human interpretability among the different linear models by calculating a quantity similar to Shannon entropy. As illustrated in Figure 1, it is evident that model (b) is significantly more understandable to humans compared to model (a). If both models exhibit equal accuracy, then a selection of model (b) over (a) is desirable, as it gives a few actionable strategies. We solve this problem by introducing the concept of interpretation entropy for assessing the degree of human interpretability of any linear model. We show that under simple conditions, our definition of interpretation entropy addresses the shortcomings of complexity-based quantification.
Furthermore, we view the overall problem of AI model explanation from the lens of classical thermodynamics.[26] It is known in thermodynamics that the equilibrium state of a system is characterized by a minimum in its Helmholtz Free Energy . Here , and represent the internal energy and entropy respectively of a system with a fixed number of particles at constant temperature and volume . Similarly, we set up a formalism in this work where the optimality of an explanation () is assessed as a trade-off between its unfaithfulness () to the underlying ground truth, and interpretation entropy (). Similar to and in classical thermodynamics, in our formalism and depend monotonically on each other. The strength of this trade-off can be tuned to identify the most stable explanation using a parameter , which plays a role similar to thermodynamic temperature . For any choice of , is then guaranteed to have exactly one minimum characterized by a pair of values under certain conditions.
Results
Interpretation Unfaithfulness (U) for surrogate model construction
Our starting point is some given dataset and corresponding predictions coming from a black-box model. For a particular element , we seek explanations that are as human interpretable as possible while also being as faithful as possible to in the vicinity of . We aim to address this problem of explaining by developing a linear approximation instead, which is more interpretable due to its linear construction. Specifically, we formulate as a linear combination of an ordered set of representative features, . Typically, these features are domain dependent e.g., one-hot encoded superpixels for an image, keywords for text, and standardized values for tabular data. We demonstrate this in Eq. (1) below, where represents the linear approximation, is a constant, and comes from an ordered set of feature coefficients, .
Let’s consider a specific problem where is a high-dimensional instance, and is a black-box model prediction, for which an explanation is needed. We first generate a neighborhood of samples by randomly perturbing the high-dimensional input space.11 A detailed discussion of neighborhood generation is provided in Sec. IV. Afterwards, the black-box predictions associated with each sample in the neighborhood are obtained. Subsequently, a linear, local surrogate model is constructed using linear regression with the loss function defined in Eq. (2).
Here is a Gaussian similarity measure, where is the distance between the explanation instance and a neighborhood sample . In previous surrogate model construction approaches,[15] Euclidean distance in the continuous input feature space has been the typical choice for . However, if the input space has several correlated or redundant features, a similarity measure based on Euclidean distance can be misleading.[44, 45] TERP addresses this problem by computing a one-dimensional (1-d) projection of the neighborhood using linear discriminant analysis[46] (LDA), which removes redundancy and produces more accurate similarity. Such a projection encourages the formation of two clusters in a 1-d space, corresponding to in-class and not in-class data points respectively by minimizing within-class variance and maximizing between-class distances. Since the projected space is one-dimensional, there is no need to tune the hyperparameter, in as might be necessary in established methods and we can set . Advantages of LDA-based similarity have been highlighted for a practical problem in Sec. II D Figure 3(e-Figure 3g).

Figure 3. Using TERP to explain VAMPnets for molecular dynamics simulations of alanine dipeptide in vacuum. (a) Representative conformational states of alanine dipeptide labelled I, II, III. Projected converged states are highlighted in three different colors as obtained by VAMPnets along (b) dihedral angles. 713 different configurations are chosen for TERP and the first, and second dominant features are highlighted using colored (). (d) Relative feature importance of a specific example point A. (e) High dimensional neighborhood data projected onto 1-d using LDA for improved similarity measure. Binarizing the class prediction probabilities of the neighborhood using a threshold of 0.5 results in explanation and not explanation classes respectively. The LDA projection separates the two regimes of prediction probability, showing meaningful projection. Average similarity error, (Eq. (9)) per datapoint for (f) Euclidean, and (g) LDA based similarity respectively. Comparison between (f) and (g) shows minimal error for LDA based similarity, specifically demonstrated for an input space constructed from the four dihedral angles plus one pure noise, four pure noise, and four correlated features with partial noise respectively. The input space for no actual data and four pure noise features in (f) establishes a baseline, showing that the Euclidean similarity will include significant error even when one redundant feature is included. All the calculations were performed in 100 independent trials to appropriately examine the effects.
Next, we introduce a meaningful unfaithfulness measure () of the generated interpretation, computed from the correlation coefficient between linear model predictions () and ground truth (). For any interpretation, and thus interpretation unfaithfulness is bounded,
Using these definitions, we implement a forward feature selection scheme by first constructing linear models, each with non-zero coefficients. We use Eq. (3) to identify the feature responsible for the lowest . Here, the superscript highlights that was calculated for a model with non-zero coefficients. We will follow this notation for other relevant quantities throughout this manuscript.
Afterwards, the selected feature is propagated to identify the best set of two features resulting in the lowest , and the scheme is continued until is computed. Since a model with non-zero coefficients will be less or at best equally unfaithful as a model with non-zero coefficients defined in Eq. (1), it can be observed that monotonically decreases with . The overall scheme generates distinct interpretations as goes from 1 to .
Interpretation Entropy (S) for model selection
After identifying interpretations, our goal is to determine the optimal interpretation from this family of models. At this point, we introduce the definition of interpretation entropy for quantifying the degree of human interpretability of any linear model. Given a linear model with an ordered set of feature coefficients among which are non-zero, we can define , where . The interpretation entropy is then defined as:
Here the subscript indicates that was calculated for a model with non-zero coefficients. It is easy to see that satisfies the properties of a probability distribution. Specifically, and .
Lemma 01. Similar to the concept of self-information/surprisal in information theory, the negative logarithm of from a fitted linear model can be defined as the self-interpretability penalty of that feature. Interpretation entropy is then computed as the expectation value of self-interpretability penalty of all the features as shown in Eq. (5). Using Jensen’s inequality it is easy to show that has an upper limit of and we can normalize the definition so that is bounded between .
This functional form of interpretability penalty i.e., interpretation entropy () encourages low values for a sharply peaked distribution of fitted weights indicating high human interpretability and vice-versa. Furthermore, if the features are independent, has two very interesting properties expressed in the theorems below. The proofs of these two crucial theorems are provided in supporting information (SI).
Theorem 01: is a monotonically increasing function of the number of features ().
Theorem 02: monotonically increases as decreases.
Free Energy () for optimal explanation
For an interpretation with non-zero coefficients, we now define free energy () as a trade-off between , and tunable by a parameter :
By writing an expression (Eq. (7)) for the stationary value, , we can define characteristic temperatures at each . Essentially, is a measure of change in unfaithfulness per unit change in interpretation entropy for a model with non-zero coefficients. This closely resembles the definition of thermodynamic temperature which is defined as the derivative of internal energy with respect to entropy. Afterwards, we identify the interpretation with non-zero coefficients that minimizes as the optimal interpretation since it is guaranteed that will preserve the lowest minimum among the set within the widest range of temperatures. Finally, we calculate optimal temperature, (any value within is equally valid since the optimal interpretation itself does not change) and generate the explanation as weights of this model. All vs. plots shown in this manuscript are created using this definition of optimal temperature.

Figure 2. Illustrative example highlighting properties of free energy , unfaithfulness , and interpretation entropy . (a) Strength of contribution to can be tuned using . vs. plots for three different are shown resulting in a minimum at respectively. (b) vs. remains unaffected by . (c) vs. plot shows that the strength of the trade-off can be tuned by .
Thus,
This is again reminiscent of classical thermodynamics where a system’s equilibrium configuration will in general vary with temperature but the coarse-grained metastable state description remains robust over a well-defined range of temperatures. In our framework, when , is minimized at interpretation or the model that maximizes unfaithfulness and completely ignores entropy. As is increased from zero, interpretation entropy contributes more to . Here, is a measure of the stability of the non-zero coefficient interpretation. The complete TERP protocol is summarised in Algorithm 1.
| Algorithm 1 TERP protocol |
| 1: Generate neighborhood data by perturbing input features. Obtain associated black-box predictions. |
| 2: Normalize, and then compute the similarity of the neighborhood samples using linear discriminant analysis (LDA). |
| 3: Using ridge regression[47] construct linear, surrogate models for all possible combinations of features at a specific . |
| 4: Implement forward feature selection by choosing the model with the lowest at a specific . |
| 5: Compute corresponding to all the chosen interpretations |
| 6: Obtain the optimal explanation by computing characteristic of the models and identifying minimum . |
Table 1.
Application to AI-augmented MD: VAMPnets
Variational approach for markov processes (VAMPnets) is a popular technique for analyzing molecular dynamics (MD) trajectories.[27] VAMPnets can be used to featurize, transform inputs to a lower dimensional representation, and construct a markov state model[48] in an automated manner by maximizing the so called VAMP score. Detailed discussion of VAMPnets theory and parameters are provided in Sec. IV B.
In this work, we trained a VAMPnets model on a standard toy system: alanine dipeptide in vacuum. An 8-dimensional input space with sines and cosines of all the dihedral angles was constructed and passed to VAMPnets. VAMPnets was able to identify three metastable states I, II, and III as shown in Figure 3 (b,Figure 3c).
To explain VAMPnets model predictions using TERP, we picked 713 different configurations, some of which are near different transition states. To quantify data points as being a transition state, we use the criterion that the prediction probability for both classes should be higher than a threshold of 0.4. From a physics perspective, the behavior of such molecular systems near the transition states is a very pertinent question. Additionally, class prediction probability is the most sensitive at the transition state and if our method generates a meaningful local neighborhood it will include a broad distribution of probabilities resulting in highly accurate approximations to the black-box behavior. Thus, a correct analysis of the transition state ensemble will validate our similarity metric and overall neighborhood generation scheme.
We generated 5000 neighborhood samples for each configuration and performed TERP by following Algorithm 1. In Figure 3 (b,Figure 3c) we highlight the first, and second most dominant features using colored stars () identified by TERP for all the 713 configurations. The generated explanations are robust and TERP identified various regions where different dihedral angles are relevant to predictions. The results are in agreement with previous literature, e.g, the relevance of dihedral angle at the transition state between I and III as reported by Chandler et al.[49]. Also, the results intuitively make sense, e.g, we see the VAMPnets state definitions change rapidly near and TERP learned that is the most dominant feature in that region. This shows that VAMPnets worked here for the correct reasons and can be trusted. In Figure 3 (d,e,f) we show TERP results for a specific configuration for which non-zero model resulted in optimal interpretation. Figure 3 (f) clearly shows that is minimized at and the average of is taken as the optimal temperature for calculating (note the minimum at ) using Eq. (8). Additional implementation details are provided in Sec. IV B.
Next, we demonstrate the advantages of LDA based similarity measure. Figure 3(e) shows that the LDA projection successfully generated two clusters of datapoints belonging to the in-explanation and not-in-explanation classes respectively. These well separated clusters help in computing meaningful and improved distance measure . In Figure 3(f,Figure 3g) we illustrate the robustness of an LDA implementation against noisy and correlated features and compare results with Euclidean similarity implementation. We generate pure white noise by drawing samples from a normal distribution and generate correlated data by taking (e.g. ), where are standardized features from the actual data. As shown in Figure 3(f,Figure 3g), we construct synthetic neighborhoods by combining actual data from the four dihedral angles and adding one pure noise, four pure noise, four correlated features respectively. Since the synthetic features do not contain any information, their addition should not change similarity. Thus we can compare the robustness of a measure by computing average change in similarity per datapoint squared which we call similarity error, as
belongs to the in-explanation and not-in-explanation classes respectively. These well separated clusters help in computing meaningful and improved distance measure . In Figure 3(f,Figure 3g) we illustrate the robustness of an LDA implementation against noisy and correlated features and compare results with Euclidean similarity implementation. We generate pure white noise by drawing samples from a normal distribution and generate correlated data by taking (e.g. ), where are standardized features from the actual data. As shown in Figure 3(f,Figure 3g), we construct synthetic neighborhoods by combining actual data from the four dihedral angles and adding one pure noise, four pure noise, four correlated features respectively. Since the synthetic features do not contain any information, their addition should not change similarity. Thus we can compare the robustness of a measure by computing average change in similarity per datapoint squared which we call similarity error, as shown in Eq. (9).
Here, the superscripts and represent similarities corresponding to the original and synthetic datapoints respectively. From Figure 3(g,Figure 3h) we can see that LDA based similarity performs significantly better in 100 independent trials compared to Euclidean similarity. On the other hand, the addition of one pure noise introduces significant similarity error for Euclidean measure. Thus we can conclude that adopting LDA over Euclidean similarity measure will produce significantly improved explanation.
In this section, we demonstrated the applicability of TERP for probing black-box models designed to analyze time-series data coming from MD simulations. In addition to assigning confidence to these models, TERP can be used to extract valuable insights (relevant degrees of freedom) learned by the model. In the future, we expect an increased adoption of TERP-like methods in the domain of AI-enhanced MD simulations for investigating conformational dynamics, nucleation, target-drug interactions, and other relevant molecular phenomena.
Application to Image classification: Vision Transformers (ViTs)
Transformers are a type of machine learning model characterized by the presence of self-attention layers and are commonly used in natural language processing (NLP) tasks.[51] The more recently proposed Vision transformers (ViTs)[28] aim to directly apply the transformer architecture to image data, eliminating the need for convolutional layers and have become a popular choice in computer vision. Per construction, ViTs are black-box models and because of their practical usage, it is desirable to employ an explanation scheme to validate their predictions before deploying them.
ViTs operate by segmenting input images into smaller patches, treating each patch as a token similar to words in NLP. These patches are then embedded (patch-embeddings) and passed to the transformer layers conducting self-attention and feedforward operations. Such a design allows ViTs to capture long-range spatial dependencies within images and learn meaningful representations. Interestingly, ViTs are known to perform poorly with limited training data but with sufficiently large datasets, ViTs have been shown to outperform convolutional layer-based models. Thus a typical ViT implementation includes two stages: first a large dataset is used to learn meaningful representation and pre-train a transferable model, followed by fine-tuning for specific tasks.
In this work, we employ a ViT pre-trained on the ImageNet-21k dataset from the authors[28, 52, 53] and then fine-tune the model for predicting human facial attributes by training on the publicly available large-scale Celeb-Faces Attributes (CelebA)[54] dataset. CelebA is a large collection of 202,599 human facial images and each image is labeled with 40 different attributes (e.g, ‘Smiling’, ‘Eyeglasses’, ‘Male’, etc.). During training, input images are converted into 16x16 pixel patches resulting in a total of 196 patches for each CelebA image (224x224 pixel) depicted in Figure 4 (b). Other details of the architecture and training procedure are provided in Sec. IV C.

Figure 4. Using TERP to explain and check the reliability of a ViT trained on CelebA dataset. (a) ViT predicts the presence of ’Eyeglasses’ in this image with a probability of 0.998. (b) Superpixel definitions for the test image following the 16x16 pixel definition of ViT patches. TERP results showcasing (c) , (d) , (e) , and (f) as a function of , (g) corresponding TERP explanation. We can see the maximal drop in happens when going from to . By defining the optimal temperature as discussed in Sec. II C a minimum in is observed at . Panels (h-j) show sanity checks[50] i.e, the result of an AI explanation scheme should be sensitive under model parameter randomization (h,i) and data randomization (j). (k) Saliency map results as baseline explanation for ‘Eyeglasses’ prediction. Red color highlights pixels with high absolute values of the class probability gradient across RGB channels. High gradient at pixels not relevant to ‘Eyeglasses’ shows limitation of saliency map explanation. (i,j) Shows TERP and saliency map explanations for the class ‘Male’. , , , and as a function of plots are provided in the SI.
To explain the ViT prediction ‘Eyeglasses’ (prediction probability of 0.998) for the image shown in Figure 4 (a) using TERP, we first construct human-understandable representative features by dividing the image into 196 superpixels (collection of pixels) corresponding to the 196 ViT patches as shown in Figure 4 (b). Afterwards, a neighborhood of perturbed images was generated by averaging the RGB color of randomly chosen superpixels following the scheme outlined in Sec. (IV A). Figure 4 (c-Figure 4f) show , , , and as functions of after implementing the TERP protocol (Algorithm 1). Thus, TERP explanation enables us to conclude that the ViT prediction of ‘Eyeglasses’ was made for the correct reasons. The optimal TERP explanation shown in Figure 4 (g) appears at , due to the maximal decrease in as is increased from 2 to 3. Using Eq. (7),(8) is calculated and a minimum occurs at .
To establish that TERP indeed takes both the input data and the black-box model into account when generating explanations we subject our protocol to the sanity tests developed by Adebayo et al.50. We achieve this by taking the fine-tuned ViT model and randomizing the model parameters in a top-to-bottom cascading fashion following their work and obtain corrupted models. Specifically, we randomize all parameters of ViT blocks 11 – 9, and blocks 11 – 3 respectively to obtain two corrupt models. TERP explanations for ‘Eyeglasses’ for these two models are shown in Figure 4 (h-Figure 4i). Plots showing , , , and as functions of for these models are provided in the SI. Here, the idea is that, due to randomization the explanation will not match the ground truth but a good AI explanation scheme should be sensitive to this randomization test and produce different explanations from the fully trained model. Similarly, we implemented the data randomization test (Figure 4 (j)) proposed in the same work, where the labels of the training data are randomized prior to training and a new ViT is trained (training details provided in the SI) using the corrupted data. Again, the results of an AI explanation method should be sensitive to this randomization. From the TERP explanations shown in panels (h-i), we conclude TERP passes both randomization tests.
Furthermore, we compute saliency map (Sec. IV C) explanations for ‘Eyeglasses’ prediction as baseline. As shown in Figure 4 (k), we see the limitations of the saliency explanation e.g, a lot of pixels irrelevant to ‘Eyeglasses’ are detected to have high absolute values of the probability gradient across the RGB channels. This is not surprising since saliency maps are known to detect color changes, object edges, and other high-level features instead of learning a relationship between model inputs and class prediction.[50] This analysis serves as a baseline and helps to put TERP explanations into context. Finally, we also generate TERP and saliency map explanations for the label ’Male’ as shown in Figure 4 (l,m). A more detailed analysis of TERP explanations for this prediction is provided in the SI. We would also like to emphasize that in addition to helping to decide whether an image classifier can be trusted or not, TERP should be useful for uncovering underlying patterns and biases in the training data.
Application to Text classification: Attention-based Bidirectional Long Short-Term Memory (Att-BLSTM)
Classification tasks in natural language processing (NLP) involve identifying semantic relations between units appearing at distant locations of in a block of text. This challenging problem is known as relation classification and models based on long short-term memory (LSTM)[55], gated recurrent unit (GRU)[56], transformers[51] have been very successful in addressing such problems. In this work, we look at the widely used attention-based bidirectional long short-term memory[29] (Att-BLSTM) classifier and apply TERP to explain its predictions. First, we trained an Att-BLSTM model on Antonio Gulli’s (AG’s) news corpus[57], which is a large collection of more than 1 million news articles curated from more than 2,000 news sources. The labels associated with each news article in the dataset indicate the section of the news source (e.g, World, Sports, Business, or Science and technology) that the news was published in. Afterwards, we employed the trained model and obtained prediction for a story titled “AI predicts protein structures” published in ‘Nature’s biggest news stories of 2022’.[58]
To implement TERP for probing a black-box prediction involving text input (sequence of sentences), first the text is passed through a tokenizer (nltk)[59] which generates a dictionary of words/phrases contained in that text. These words are the representative features to be used in TERP. Afterwards, a neighborhood of the perturbed text is generated by randomly choosing and removing all instances of different words from the text. TERP processes the neighborhood as numerical values for linear model construction by creating a one-hot-encoded matrix where the columns represent the presence or absence of the different words in a perturbed text.
As a specific instance, the Att-BLSTM classifier predicted that the story titled “AI predicts protein structures” is about Science and Technology and we implemented TERP to generate the optimal explanation behind this prediction as shown in Figure 5. We see that the most influential keywords are ’species’, and ’science’ which gives confidence that Att-BLSTM model was able to classify the news story for the correct reasons.

Figure 5. Using TERP to explain and check the reliability of Att-BLSTM model trained on AG’s news corpus to predict the news story titled ”AI predicts protein structures”. (a) relative feature importance of the two most influential keywords: ‘science’, and ‘species’ as identified by TERP, (b) , (c) , (d) , (e) vs. plots showing the optimal explanation appears at , due to the maximum drop in as is increased from 1 to 2.
Discussion
The widespread adoption of AI-based black-box models has become a standard practice across various fields due to their ability to be deployed without requiring an in-depth understanding of the underlying processes. However, this advantage also poses challenges regarding trustworthiness and the explanation of AI models. In this study, we have introduced a novel framework inspired by thermodynamics to create interpretable representations of complex black-box models. Our objective was to find representations that minimize discrepancies from the true model while remaining highly interpretable to humans using a concept similar to the energy-entropy trade-off. Furthermore, the concept of interpretation entropy introduced in this work has the potential to be useful in more general human interpretability-based model selection from any family of linear models.
We showcased the effectiveness of this approach in various AI applications, including image classification, text analysis, and molecular simulations. While several methods[16, 19, 60, 61] have been proposed to address AI interpretability in the past, only a handful, such as Ref. [62, 63, 64], have been utilized to elucidate molecular simulations. Importantly, our work marks one of the pioneering applications of interpretability techniques in the rapidly evolving field of AI-enhanced molecular dynamics.
Recent applications of our framework (TERP), have been instrumental in uncovering key mechanisms behind crystal nucleation[65] and hydrophobic ligand dissociation.[66] Given the critical role of molecular sciences in uncovering chemical reaction pathways,[67] understanding disease mechanisms,[68] designing effective drugs,[69] and numerous other vital areas, it is crucial to ensure accurate analysis, as errors in black-box models can have significant financial and public health implications. TERP should provide practitioners of molecular sciences a way to explain these black-box models on a footing made rigorous through simple yet powerful parallels with the field of thermodynamics.
Methods
Neighborhood generation
We generated neighborhood data around an instance by first drawing numbers from a uniform distribution associated with each feature, . The feature is perturbed if , otherwise the feature is kept unchanged. Once a feature is chosen for perturbation, the specific scheme for obtaining perturbed values depend on the corresponding data type.
For tabular data if a feature is continuous, it is updated by where is the standard deviation of the feature in the training data and is a small noise drawn from a gaussian distribution. For categorical data, feature value is updated by , where is sampled from the training data. For text, an instance is first converted into tokens,[70] which are considered as features. If a token is chosen for perturbation by following the scheme described above, it is replaced by a new token sampled from training data. For images, superpixels are defined and if chosen for perturbation, are updated by averaging the colors of all the pixels within that particular super-pixel.
AI-augmented MD method: VAMPnets
The molecular system for alanine dipeptide in vacuum was parametrized using the forcefield CHARMM36m[71] and prepared using CHARMM-GUI[72]. A 100 ns MD simulation of alanine dipeptide in vacuum at 450 K temperature and 1 atm pressure was performed using Nose-Hoover thermostat and Parrinello-Rahman barostat[73, 74] in GROMACS.[75]
A VAMPnets[27] deep neural network was constructed from two identical artificial neural network lobes, that take trajectory order parameters (OPs) at time steps and respectively as inputs. The input data was passed through several layers of neurons and finally a VAMP-2 score was calculated by merging results from outputs of both lobes. The neural network model parameters were tuned in successive iterations that maximize the VAMP-2 score. In this way, a markov state model at a specific lagtime can be learnt that describes the slow processes of interest.
In this work, the VAMPnets implementation was performed using the PyEMMA76 (2.5) and Deeptime77 0.4.2 python libraries by constructing the neural network architecture depicted in Supporting Fig. 2. Other training hyperparameters are: , , .
Image classification: Vision Transformers (ViTs)
Large-scale CelebFaces Attributes (CelebA) Dataset54 contains 202,599 celebrity images, each annotated with 40 binary attributes. CelebA offers the dataset in two different formats: (a) actual raw images, (b) processed data with aligned facial images. In this work, we employed the latter and divided the dataset into training, validation sets with a ratio of 50 : 50. The training data was then used to train a ViT model.
The model was trained until validation metrics (f1 score) did not improve for 5 consecutive epochs using a learning rate of 0.00001. The model with the highest validation metric was saved as the trained model.
Training and inference using ViT was implemented using pytorch-lightning 1.5 and python 3.9. The pre-trained ViT model was pulled from the timm python library. For saliency analysis, the absolute values of the gradients of prediction probabilities with respect to input pixels were calculated using the backward() method of pytorch during a backward pass.
Text classification: Attention-based Bidirectional Long Short-Term Memory (Att-BLSTM)
In this work, we employed Python implementation of Att-BLSTM29 obtained from https://github.com/Renovamen/Text-Classification with pre-trained GloVe word embedding. Att-BLSTM model was trained on Antonio Gulli’s (AG’s) news corpus78 for 10 epochs, finally reaching a validation accuracy of 92.0%.
Competing Interests
The authors have no competing interests to declare.
Author Contributions
P.T. and S.M. designed research; S.M. performed research; S.M. analyzed data; S.M. and P.T. wrote the paper.
Acknowledgments
This work was supported by the National Science Foundation, grant no. CHE-2044165 (S.M. and P.T.). S.M. was also supported through NCI-UMD Partnership for Integrative Cancer Research. The authors thank Deepthought2, MARCC, and XSEDE (projects CHE180007P and CHE180027P) for the computational resources used in this work. P.T. is an investigator at the University of Maryland-Institute for Health Computing, which is supported by funding from Montgomery County, Maryland and The University of Maryland Strategic Partnership: MPowering the State, a formal collaboration between the University of Maryland, College Park, and the University of Maryland, Baltimore. P.T. was an Alfred P. Sloan Foundation fellow during the preparation of this manuscript. The authors thank Dr. Eric Beyerle and Dr. Luke Evans for insightful discussions.
Data availability statement
The data that support the findings of this study are openly available. The AG’s news corpus dataset was obtained from Ref. 79, and CelebA dataset from Ref. 80 in accordance with the Terms of Service of the respective web resources. Data generation details for the molecular dynamics of alanine dipeptide are provided in Sec. IVB and the trajectory is available at figshare.com/articles/dataset/Black-box_models_for_TER P_interpretation/24475003. Source data is provided with this paper as additional material.
Code availability statement
Python implementation of TERP is available at github.com/tiwarylab/TERP, and the three black-box models trained on specific systems studied in this work are available at figshare.com/articles/dataset/Black-box_models_for_TER P_interpretation/24475003.
References
- [1]V. Dhar, “Data science and prediction,” Communications of the ACM 56, 64–73 (2013).
- [2]S. Shalev-Shwartz and S. Ben-David, Understanding machine learning: From theory to algorithms (Cambridge university press, 2014).
- [3]Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature 521, 436–444 (2015).
- [4]A. Davies, P. Veličković, L. Buesing, S. Blackwell, D. Zheng, N. Tomašev, R. Tanburn, P. Battaglia, C. Blundell, A. Juhász, et al., “Advancing mathematics by guiding human intuition with ai,” Nature 600, 70–74 (2021).DOI
- [5]G. Carleo, I. Cirac, K. Cramer, L. Daudet, M. Schuld, N. Tishby, L. Vogt-Maranto, and L. Zdeborová, “Machine learning and the physical sciences,” Reviews of Modern Physics 91, 045002 (2019).
- [6]A. C. Mater and M. L. Coote, “Deep learning in chemistry,” Journal of chemical information and modeling 59, 2545–2559 (2019).
- [7]P. Hamet and J. Tremblay, “Artificial intelligence in medicine,” Metabolism 69, S36–S40 (2017).
- [8]P. Baldi and S. Brunak, Bioinformatics: the machine learning approach (MIT press, 2001).
- [9]S. L. Brunton and J. N. Kutz, “Machine learning, dynamical systems, and control,” Cambridge University Press, 2022.
- [10]C. Rudin, “Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead,” Nature Machine Intelligence 1, 206–215 (2019).
- [11]C. Molnar, Interpretable machine learning (Lulu. com, 2020).
- [12]P. Linardatos, V. Papastefanopoulos, and S. Kotsiantis, “Explainable ai: A review of machine learning interpretability methods,” Entropy 23, 18 (2020).
- [13]A. B. Arrieta, N. Díaz-Rodríguez, J. Del Ser, A. Bennetot, S. Tabik, A. Barbado, S. García, S. Gil-López, D. Molina, R. Benjamins, et al., “Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai,” Information fusion 58, 82–115 (2020).
- [14]P. P. Angelov, E. A. Soares, R. Jiang, N. I. Arnold, and P. M. Atkinson, “Explainable artificial intelligence: an analytical review,” Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 11, e1424 (2021).
- [15]M. T. Ribeiro, S. Singh, and C. Guestrin, “‘why should i trust you?’” explaining the predictions of any classifier,” in Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining (2016) pp. 1135–1144.
- [16]S. M. Lundberg and S.-I. Lee, “A unified approach to interpreting model predictions,” Advances in neural information processing systems 30 (2017).
- [17]G. Montavon, A. Binder, S. Lapuschkin, W. Samek, and K.-R. Müller, “Layer-wise relevance propagation: an overview,” Explainable AI: interpreting, explaining and visualizing deep learning, 193–209 (2019).DOI
- [18]J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller, “Striving for simplicity: The all convolutional net,” arXiv preprint arXiv:1412.6806 (2014).DOI
- [19]M. Sundararajan, A. Taly, and Q. Yan, “Axiomatic attribution for deep networks,” in International conference on machine learning (PMLR, 2017) pp. 3319–3328.
- [20]H. Akaike, “A new look at the statistical model identification,” IEEE transactions on automatic control 19, 716–723 (1974).
- [21]G. Schwarz, “Estimating the dimension of a model,” The annals of statistics, 461–464 (1978).
- [22]G. A. Miller, “The magical number seven, plus or minus two: Some limits on our capacity for processing information.” Psychological review 63, 81 (1956).
- [23]G. Gigerenzer and H. Brighton, “Homo heuristicus: Why biased minds make better inferences,” topics in cognitive science, 1 (1), 107–143,” (2009).DOI
- [24]J. Chang, S. Gerrish, C. Wang, J. Boyd-Graber, and D. Blei, “Reading tea leaves: How humans interpret topic models,” Advances in neural information processing systems 22 (2009).
- [25]P. Bromiley, N. Thacker, and E. Bouhova-Thacker, “Shannon entropy, renyi entropy, and information,” Statistics and Inf. Series (2004-004) 9, 2–8 (2004).
- [26]H. B. Callen, “Thermodynamics and an introduction to thermostatistics,” (1985).
- [27]A. Mardt, L. Pasquali, H. Wu, and F. Noé, “Vampnets for deep learning of molecular kinetics,” Nature communications 9, 1–11 (2018).
- [28]A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929 (2020).arxiv.org/abs/2010.11929
- [29]P. Zhou, W. Shi, J. Tian, Z. Qi, B. Li, H. Hao, and B. Xu, “Attention-based bidirectional long short-term memory networks for relation classification,” in Proceedings of the 54th annual meeting of the association for computational linguistics (volume 2: Short papers) (2016) pp. 207–212.
- [30]A. Ma and A. R. Dinner, “Automatic method for identifying reaction coordinates in complex systems,” The Journal of Physical Chemistry B 109, 6769–6779 (2005).
- [31]Y. Wang, J. M. L. Ribeiro, and P. Tiwary, “Machine learning approaches for analyzing and enhancing molecular dynamics simulations,” Current opinion in structural biology 61, 139–145 (2020).
- [32]D. Frenkel and B. Smit, Understanding molecular simulation: from algorithms to applications, Vol. 1 (Elsevier, 2001).
- [33]S. Doerr, M. Majewski, A. Pérez, A. Krämmer, C. Clementi, F. Noe, T. Giorgino, and G. De Fabritiis, “Torchmd: A deep learning framework for molecular simulations,” Journal of chemical theory and computation 17, 2355–2363 (2021).
- [34]J. Han, L. Zhang, R. Car, et al., “Deep potential: A general representation of a many-body potential energy surface,” arXiv preprint arXiv:1707.01478 (2017).DOI
- [35]X. Gao, F. Ramezanpour, O. Isayev, J. S. Smith, and A. E. Roitberg, “Torchani: a free and open source pytorch-based deep learning implementation of the ani neural network potentials,” Journal of chemical information and modeling 60, 3408–3415 (2020).DOI
- [36]D. Wang and P. Tiwary, “State predictive information bottleneck,” The Journal of Chemical Physics 154, 134111 (2021).
- [37]E. R. Beyerle, S. Mehdi, and P. Tiwary, “Quantifying energetic and entropic pathways in molecular systems,” The Journal of Physical Chemistry B (2022).
- [38]J. M. L. Ribeiro, P. Bravo, Y. Wang, and P. Tiwary, “Reweighted autoencoded variational bayes for enhanced sampling (rave),” The Journal of chemical physics 149, 072301 (2018).
- [39]E. Vanden-Eijnden, “Transition path theory,” An introduction to Markov state models and their application to long timescale molecular simulation, 91–100 (2014).
- [40]Z. Smith, P. Ravindra, Y. Wang, R. Cooley, and P. Tiwary, “Discovering protein conformational flexibility through artificial-intelligence-aided molecular dynamics,” The Journal of Physical Chemistry B 124, 8221–8229 (2020).DOI
- [41]S. Mehdi, D. Wang, S. Pant, and P. Tiwary, “Accelerating all-atom simulations and gaining mechanistic understanding of biophysical systems through state predictive information bottleneck,” Journal of Chemical Theory and Computation 18, 3231–3238 (2022).
- [42]E. R. Beyerle, Z. Zou, and P. Tiwary, “Recent advances in describing and driving crystal nucleation using machine learning and artificial intelligence,” Current Opinion in Solid State and Materials Science 27, 101093 (2023).DOI
- [43]S. Mehdi, Z. Smith, L. Herron, Z. Zou, and P. Tiwary, “Enhanced sampling with machine learning,” Annual Review of Physical Chemistry 75 (2024).
- [44]M. Karagiannopoulos, D. Anyfantis, S. Kotsiantis, and P. Pintelas, “Feature selection for regression problems,” Educational Software Development Laboratory, Department of Mathematics, University of Patras, Greece (2004).
- [45]K.-Y. Liang and S. L. Zeger, “Regression analysis for correlated data,” Annual review of public health 14, 43–68 (1993).
- [46]A. J. Izenman and A. J. Izenman, “Linear discriminant analysis,” Modern multivariate statistical techniques: regression, classification, and manifold learning, 237–280 (2008).
- [47]A. E. Hoerl and R. W. Kennard, “Ridge regression: applications to nonorthogonal problems,” Technometrics 12, 69–82 (1970).
- [48]G. R. Bowman, V. S. Pande, and F. Noé, An introduction to Markov state models and their application to long timescale molecular simulation, Vol. 797 (Springer Science & Business Media, 2013).
- [49]P. G. Bolhuis, C. Dellago, and D. Chandler, “Reaction coordinates of biomolecular isomerization,” Proceedings of the National Academy of Sciences 97, 5877–5882 (2000).
- [50]J. Adebayo, J. Gilmer, M. Muelly, I. Goodfellow, M. Hardt, and B. Kim, “Sanity checks for saliency maps,” Advances in neural information processing systems 31 (2018).arxiv.org/abs/1810.03292
- [51]A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems 30 (2017).arxiv.org/abs/1706.03762
- [52]A. Steiner, A. Kolesnikov, X. Zhai, R. Wightman, J. Uszkoreit, and L. Beyer, “How to train your vit? data, augmentation, and regularization in vision transformers,” arXiv preprint arXiv:2106.10270 (2021).
- [53]R. Wightman, “Pytorch image models,” https://github.com/huggingface/pytorch-image-models (2019).
- [54]Z. Liu, P. Luo, X. Wang, and X. Tang, “Large-scale celebfaces attributes (celeba) dataset,” Retrieved August 15, 11 (2018).
- [55]Y. Yu, X. Si, C. Hu, and J. Zhang, “A review of recurrent neural networks: Lstm cells and network architectures,” Neural computation 31, 1235–1270 (2019).DOI
- [56]J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, “Empirical evaluation of gated recurrent neural networks on sequence modeling,” arXiv preprint arXiv:1412.3555 (2014).arxiv.org/abs/1412.3555
- [57]A. Gulli, “The anatomy of a news search engine,” in Special interest tracks and posters of the 14th international conference on World Wide Web (2005) pp. 880–881.
- [58]“Nature’s biggest news stories of 2022,” (2022).DOI
- [59]N. Hardeniya, J. Perkins, D. Chopra, N. Joshi, and I. Mathur, Natural language processing: python and NLTK (Packt Publishing Ltd., 2016).
- [60]A. Fisher, C. Rudin, and F. Dominici, “All models are wrong, but many are useful: Learning a variable’s importance by studying an entire class of prediction models simultaneously.” J. Mach. Learn. Res. 20, 1–81 (2019).
- [61]S. Wachter, B. Mittelstadt, and C. Russell, “Counterfactual explanations without opening the black box: Automated decisions and the gdpr,” Harv. JL & Tech. 31, 841 (2017).arxiv.org/abs/1711.00399
- [62]G. P. Wellawatte, A. Seshadri, and A. D. White, “Model agnostic generation of counterfactual explanations for molecules,” Chemical science 13, 3697–3705 (2022).DOI
- [63]T. Kikutsuji, Y. Mori, K.-I. Okazaki, T. Mori, K. Kim, and N. Matubayasi, “Explaining reaction coordinates of alanine dipeptide isomerization obtained from deep neural networks using explainable artificial intelligence (xai),” The Journal of Chemical Physics 156, 154108 (2022).
- [64]O. Fleetwood, M. A. Kasimova, A. M. Westerlund, and L. Delemotte, “Molecular insights from conformational ensembles via machine learning,” Biophysical Journal 118, 765–780 (2020).
- [65]R. Wang, S. Mehdi, Z. Zou, and P. Tiwary, “Is the local ion density sufficient to drive nacl nucleation from the melt and aqueous solution?” The Journal of Physical Chemistry B (2024).DOI
- [66]E. R. Beyerle and P. Tiwary, “Thermodynamically optimized machine-learned reaction coordinates for hydrophobic ligand dissociation,” The Journal of Physical Chemistry B 128, 755–767 (2024).DOI
- [67]M. Yang, J. Zou, G. Wang, and S. Li, “Automatic reaction pathway search via combined molecular dynamics and coordinate driving method,” The Journal of Physical Chemistry A 121, 1351–1361 (2017).
- [68]S. A. Hollingsworth and R. O. Dror, “Molecular dynamics simulation for all,” Neuron 99, 1129–1143 (2018).DOI
- [69]H. Zhao and A. Caflisch, “Molecular dynamics in drug design,” European journal of medicinal chemistry 91, 4–14 (2015).
- [70]J. J. Webster and C. Kit, “Tokenization as the initial phase in nlp,” in COLING 1992 volume 4: The 14th international conference on computational linguistics (1992).DOI
- [71]J. Huang, S. Rauscher, G. Nawrocki, T. Ran, M. Feig, B. L. De Groot, H. Grubmüller, and A. D. MacKerell, “Charmm36m: an improved force field for folded and intrinsically disordered proteins,” Nature methods 14, 71–73 (2017).
- [72]J. Lee, X. Cheng, J. M. Swails, M. S. Yeom, P. K. Eastman, J. A. Lemkul, S. Wei, J. Buckner, J. C. Jeong, Y. Qi, et al., “Charmm-gui input generator for namd, gromacs, amber, openmm, and charmm/openmm simulations using the charmm36 additive force field,” Journal of chemical theory and computation 12, 405–413 (2016).
- [73]S. Nosé, “A unified formulation of the constant temperature molecular dynamics methods,” The Journal of chemical physics 81, 511–519 (1984).DOI
- [74]M. Parrinello and A. Rahman, “Crystal structure and pair potentials: A molecular-dynamics study,” Physical review letters 45, 1196 (1980).
- [75]D. Van Der Spoel, E. Lindahl, B. Hess, G. Groenhof, A. E. Mark, and H. J. Berendsen, “Gromacs: fast, flexible, and free,” Journal of computational chemistry 26, 1701–1718 (2005).