WorksComputer Science
HardwareDx: A Foundational Graph Model System for Datacenter Machine Repairs
Abstract
The rapid scale-up of AI infrastructure has made datacenter reliability a cornerstone of the cloud business economy. Since hardware failures in hyper-scale AI clusters can stall multi-million dollar training workloads, the downtime of individual machines can cause outsized economic losses. Minimizing machine downtime, therefore, is more important than ever. To address this, we present Hardware Diagnoser (HardwareDx), an AI-driven hardware diagnosis system that applies a graph foundation model approach to automated datacenter maintenance. It allows a single model to diagnose many platforms and accelerate onboarding of new hardware products. Deployed at production scale at Google, HardwareDx yields substantial efficiency gains across diverse compute architectures. Real-world evaluations demonstrate that it reduces production downtime by 18.3% on GPU platforms and 16.1% on TPUs. Furthermore, it significantly accelerates hardware deployment, cutting quality assurance (QA) time by 28.3% for GPUs and 25.1% for TPUs. These results establish HardwareDx as a field-tested solution for maximizing the reliability and cost efficiency of next generation AI fleets.
Introduction
State-of-the-art foundation models are typically trained using thousands to hundreds-of-thousands of interconnected machines [67]. When a single machine in a synchronized training cluster fails, the entire workload often halts, leaving thousands of other accelerators idle. In 100,000-GPU clusters, where mean time-to-failure is 15-18 minutes [32, 60], such failures create massive downtime costs. It is natural to consider leveraging models to predict failures. Indeed, many prior works [5, 13, 19, 39, 77] build algorithms and models to anticipate emerging failures. Once failures occur, reducing time-to-repair directly translates to equivalent downtime savings and efficiency, unlocking millions in value and improving velocity as researchers are able to iterate more quickly within a fixed compute budget.
Site-reliability and data-center engineers aim to mitigate this overhead by quickly detecting and isolating faulty hardware. Historically, repairs were entirely manual. As fleets scaled, engineers moved to standardized actions and simple rules, and eventually to sophisticated score-based rule systems tailored to specific hardware platforms [43, 82]. While early machine learning approaches (such as MLPs and decision forests) were explored, they failed to outperform these carefully-engineered rule systems (Appendix § 8). Upon detecting an issue, these systems isolate the faulty hardware and carry out the needed hardware or software repair. While many failures may resolve with just a single component reseat (remove and reconnect), complex comorbidity at the tail-end, such as power anomalies corrupting interconnected high-speed networking switches, can require extensive interventions. To minimize wasted time and unnecessary hardware swaps, the repair rules must be meticulously maintained, leading to fragility over time. Further, the rule set is platform and hardware-model specific [45, 48, 62, 63, 70]. Every new product introduction requires hand-curating a new set of rules for fault diagnosis and repair.
By moving from brittle, manually-maintained rule systems to a transferable graph model, HARDWAREDX introduces a practical approach for datacenter hardware diagnosis. Specifically, HARDWAREDX operates as a domain-specific graph foundation model that pretrains across different machine topologies to learn shared structural representations. The pretrained backbone transfers structural knowledge to newly deployed hardware generations with minimal data. We have a full working implementation of HARDWAREDX running for 6 months at Google across 50+ ML clusters. Our empirical evaluation shows that HARDWAREDX classifies each component as (faulty, not-faulty) with >95% precision at >95% recall. One single model can run inference across many compute platforms and across multiple tasks for each.
There are several practical benefits over a manually-updated, rule-based system [12, 25, 29, 71, 78]. First, the fault detection accuracy is higher (+12.6%), yielding 18.3% downtime reduction on GPU platforms and 16.1% downtime reduction on TPUs. Second, it allows for quick adaptation as new hardware generations are introduced to the market. By leveraging LLMs to process log lines [23, 28, 55], we eliminate the need for complex parsers and rigid schemas. This doubles development speed, allowing test engineers to define signatures for ~400 symptoms per platform in just 1 software engineering (SWE) month instead of 2. We further accelerate development by utilizing transfer learning for new platforms, enabling production-ready deployment up to four months earlier, prior to general availability of the hardware. At our scale, such velocity improvements can unlock significant additional revenue.
We summarize our contributions and originality as follows. First, we frame the machine repair problem as a node classification task operating over the physical hardware topology. Second, we present a transfer learning approach that adapts generalized representations to new hardware platforms. Third, we introduce a novel combination of learning objectives leveraging both supervised expert labels and unsupervised historical trajectories. Finally, we validate these methods through a large-scale production deployment, establishing, to our knowledge, the first published GNN-based system for AI hardware repairs. HARDWAREDX is currently deployed for six AI accelerator platforms at Google, including a new TPU platform.
Background
Data center diagnostics and repairs are highly complex operations where a primary concern is uptime for serving machines and rapid quality assurance for new datacenter builds [4, 37, 54]. From a business perspective, these operations directly impact the bottom line. Uptime for serving machines is a primary driver of cloud revenue, and rapid quality assurance directly corresponds to the server provisioning required to maintain throughput and latency targets for products [58, 83].
HardwareDx’s ability to reduce downtime leads directly to increased service availability and reduced capital expenditure on spare capacity.
IT Quality Assurance (QA) is under even bigger pressure to deliver all capacity to the customer, making long-tail repairs critical [19]. In the competitive cloud business, providers vie for the lowest “Time to Market”, rendering the repair speed of these long-tail failures even more important to ensure timely delivery of massive compute clusters.
Each machine is an expensive grouping of equipment that forms a cohesive unit. For instance, an H100 machine may consist of eight GPUs on a tray, as well as CPUs, memory chips, networking switches, a motherboard, and many types of cables to wire them together [43, 50, 82]. When a component fails, the whole machine ceases to function. This means that any workloads relying on that machine must pause as well. If those workloads are large AI training or inference runs, then many other machines could also sit idle given the bulk synchronous nature of these workloads [49].
A new machine going through setup first goes through an install process where it is assembled and the correct software is installed. Upon passing install and quality assurance, the machine is live and can start serving workloads. However, all machines will eventually fail, thus they need to be monitored. The Monitoring system continuously checks telemetry and error data, and if anything is amiss, it will take the machine offline to undergo testing and repairs.
Once offline, the testing suite runs a large battery of workloads and stress tests on the machine. This is a time-consuming process taking many hours to complete. The primary output of these tests is a binary indication of whether the machine is healthy or not, as well as data that can be used to determine what repair actions may be needed. If the machine is healthy, it can resume serving. If not healthy, it must be diagnosed. Each machine consists of over 100 Field Replaceable Units (FRUs), any of which could be faulty, and multiple possible repair actions for each one.
At this point HardwareDx ingests the data from testing and additional information such as the machine topology to output a diagnosis of the faulty component or components. Then, based on the component type repair plan and what prior actions have already been attempted, we prescribe a new repair action or set of actions for the technician to perform. The machine then goes back to the testing suite. If it continues to be unhealthy, this process continues for further iterations. Figure 1 gives a high level overview of how the repair systems and operations function.

Figure 1. Simplified flow of datacenter operations.
If HardwareDx fails to make a diagnosis due to service failures, the system falls back to using hand-coded rules. If it exceeds a certain threshold of iterations, the case goes to manual diagnosis by expert technicians.
Foundation Model and Platform Transfer
In a standard supervised learning setup, a new model has to be trained from scratch each time a new platform is released. However, with limited initial deployment volumes and long delays for diagnostics and failure modes to stabilize, it can take over a year to collect enough data to train a sufficiently strong model. To reduce such long lead times, we introduce foundation models for machine repairs, shifting from siloed, platform-specific training to a generalized model capable of solving machine repairs as soon as the hardware is released.
Rather than training a single model for each specific platform, we instead design our model to be able to handle new platforms with different hardware, tests, and component sets, as long as there is some transferable information. We can then train a model on either a single, data-rich platform or multiple platforms, and then finetune for specific platforms and tasks.
HardwareDx Model Architecture
The HardwareDx model centers around a Graph Neural Network (GNN) [8] that uses the physical hardware topology along with other diagnostic signals to predict failing components and repair actions to take. The input includes prior actions, symptoms, sensors, telemetry, and other messages, as well as the machine topology. The model then uses a combination of heterogeneous encoder layers, homogeneous GNN layers, and transformer layers, before predicting a binary faulty label for each Field Replaceable Unit (FRU). An overview of the HardwareDx architecture is illustrated in Figure 2.

Figure 2. HARDWAREDX Model Architecture. The machine topology and diagnostic signals are represented as a heterogeneous graph. Each node type is encoded according to the features it has and the topology is homogenized to create the universal graph format. This graph then goes through the GNN backbone before predictions are made for each task and component type
Machine Hardware Topology
The graph input to HARDWAREDX is primarily derived from the physical machine topology. Each component is a node in the graph representing either a Field Replaceable Unit (FRU) or a subcomponent of a FRU. For instance, the graph would contain a node for each GPU, which would each connect to subcomponents such as networking ports, power supplies ports, etc. Those subcomponents would connect to other components such as networking cables and power supplies. These components come together to form a heterogeneous graph of nodes of different component types. Figure 2 (left) illustrates a mock heterogeneous graph representation of this topology. Each machine is represented as a single, self-contained graph. Different machines are not connected, as hardware topologies and physical faults are isolated at the individual chassis/tray boundary. This also keeps graphs small and inference efficient. Each platform generally has a single physical topology, with occasionally more than one hardware variant, while the attached feature nodes (§ 3.2) vary from graph to graph in both structure and value. Appendix § 10 formally defines the graph.
Features: Sensors, Symptoms, Diagnostics
In addition to the machine topology, the model input also consists of various signals generated during stress testing. These signals include: Telemetry Data such as interconnect bandwidths, device counts, or tflops, as well as sensor readings like temperature, power, and fan-speed; Diagnostic Symptoms resulting from failing tests such as gpu-clock-speed-below-limit or timeout-error; and Error Messages parsed directly from logs.
These features are then added as nodes to the graph and attached via an edge to the corresponding component that generated them. Each feature node may have multiple actual feature values. For instance, a fan speed reading would become a node of type "sensor" with features {name: ‘fan speed’, unit: ‘hertz’, value:121.3}. Features not generated by a specific component are attached to a “software” node. Appendix § 11 lists all feature node types, their data sources, and the component types, along with feature-importance ablations.
Universal Graph Virtualization and Parameter Homogenization
The initial machine topology and diagnostic state form a highly heterogeneous graph, defined as . Here, the nodes and edges are partitioned by diverse component and relation types ( and ), and contains varying raw features ranging from floating-point telemetry to unstructured text logs.
Standard heterogeneous GNNs maintain distinct learnable weight matrices for every discrete relation type [81]. Unfortunately this direct formulation scales poorly, resulting in a combinatorial parameter explosion of [61]. In a large-scale datacenter context, this parameter bloat causes severe overfitting on sparse physical connections and results in catastrophic transfer failure when novel hardware introduces uninitialized relation types.
To resolve this bottleneck, we propose a virtualization function that maps into a unified, homogeneous graph . This homogenization results in a mirror graph containing a single node set and a single edge set. For each node of type with features , we deploy a type-specific encoder projection head (such as MLPs for continuous data or lookup tables for discrete data) to project the heterogeneous features into a shared latent space :
Each encoder is customized to the structure of the data for that node-set. For instance, if a "bandwidth" node type has a float feature for the bandwidth speed and a string indicating the flow type (e.g. "CPU-to-GPU"), then the encoder will be projecting the features into a single unified representation.
This architecture decouples feature representation from structural message-passing. By restricting the GNN to operate exclusively on the homogenized graph using shared parameters, the model acts as a strong structural regularizer. Consequently, HARDWAREDX learns generalized topological priors that seamlessly transfer to new compute platforms, requiring only the lightweight adaptation of the isolated node encoders in case new node types appear.
LLM Embeddings
Since many of the features are text strings—either parsed from log files or symptom names—it makes sense to encode these features using an LLM. Additionally, between platforms, we see slight variations in symptom names and logging for the same underlying condition. For instance, "gpu-pcie-bus-error" became "nvfdpilot-gpu-pcie-bus-error" in a platform update. A categorical embedding on symptom type would result in no foundational knowledge transfer between these features. However, with an LLM, the resulting embedding has a high similarity (0.89 cosine similarity). We use Gecko-1B-En [35] as our LLM embedding model for all string-based features to facilitate the appropriate semantic understanding from one hardware platform to another.
Graph Neural Networks & Long Range Dependencies
GNNs capture the inherent structural elements of diagnostic data. Spatial relationships provide critical diagnostic signals; for instance, multiple similar symptoms occurring across connected GPUs strongly suggest an issue with a shared upstream component. Conversely, a single isolated symptom might implicate a variety of connected devices, such as a faulty interconnect cable, defective ports, or the underlying hardware attached to that cable. Based on experimental evaluation, we selected the Graph Attention Network v2 (GATv2) [7] as our primary message-passing layer to model these local structural dependencies.
Standard Message-Passing Neural Networks (MPNNs) can struggle with long-range dependencies due to over-squashing of information [1, 3, 69]. Our model must diagnose failure syndromes spanning up to eight graph hops and integrate signals from disconnected “software” nodes with non-physical actions and software symptoms. While various solutions for handling long-range dependencies exist [21, 69], we use a Graph Transformer based solution. Instead of positional embeddings to encode local structure [15, 80], we follow Rampášek et al. [57] in interleaving Transformer layers with local MPNN layers (GATv2) to directly encode local graph topology. This design permits unconstrained, direct attention between any pair of nodes in the graph, elegantly resolving the dual challenges of long-range, multi-hop syndrome propagation and the integration of disconnected nodes. Ultimately, these design choices are based on experimental validation (§ 4.4).
Data Formulation
Let be the set of system components (FRUs) and be the set of possible repair actions, including a designated “no-action”. We leverage two distinct data sources for training: an unsupervised historical dataset and a supervised expert-labeled dataset.
Historical Diagnostic Sequences (Unsupervised): The raw telemetry data consists of episodes representing a machine’s journey through the repair process. An episode is a sequence of failed tests interspersed with repair actions, terminating when a test finally passes. We formulate this as a trajectory:
where represents the diagnostic graph and telemetry features at step , and is a binary mask indicating the specific actions applied to specific components after test . Each row sums to 1 as only one action can be prescribed to each component, and if no action is prescribed then there is a 1 in the "no-action" column. The subsequent test result acts as our binary success label, where for all steps (next test fails), and for the final step (next test passes). This yields a dataset of historical transitions .
Expert-Labeled Diagnostics (Supervised): We also develop smaller datasets of expert-annotated ground truth faults, denoted as , where is a binary vector indicating which components should be considered faulty. This data takes many months to develop for each platform as new failure modes are observed and labeled. See Appendix § 14 for details on data collection.
Multi-Task Prediction Heads and Loss Functions
HARDWAREDX employs a multi-task learning framework operating on the shared graph representations. The model branches into three distinct prediction heads for each platform, each with a tailored loss function.
Task 1: Predict Faulty. The first head isolates the diagnostic objective, outputting a binary fault prediction for each component, . This is supervised exclusively by the expert-labeled dataset . We optimize the standard binary cross-entropy (BCE) loss over all components :
Task 2: Predict Action Propensity. The second head models the general behavioral prior of interventions by predicting the propensity of specific actions being taken, regardless of their outcome. For this task, we utilize the complete historical dataset . The model predicts the probability of each action for each component, , utilizing a sigmoid activation. The loss is computed as the expected sigmoid cross-entropy over all datapoints against the observed actions taken by the technicians:
Task 3: Conditional Likelihood of Success. The final head evaluates the viability of specific actions by learning from the entirety of the historical data , encompassing both failed and successful repair attempts. The goal is to estimate the likelihood of passing the next test upon taking a set of actions, . The model outputs a dense matrix , where each entry represents the likelihood of success conditioned on applying action to component .
During training, we use the observed action matrix as a binary mask to select only the predictions corresponding to the actions actually taken by technicians. The masked predictions are evaluated against the binary next-test outcome using a sigmoid BCE loss:
The total loss optimized during training is the sum of these three components:
Inference and Prescriptive Action Selection
At inference time, HARDWAREDX functions as a prescriptive policy, mapping the current machine state to an optimal repair action. We have two approaches for inference. The primary approach leverages Task 1 () directly. The second approach leverages the outputs of Task 2 () and Task 3 () to generate a recommendation through a gated, heuristic selection strategy.
Faulty-based Inference
The faulty-based inference is based on Task 1. Under this use-case, for every component type there is a specific action plan (e.g., For CPU faults: Reseat first, Swap second, Swap Motherboard third). We set a threshold, , and any component with a predicted likelihood of being faulty greater than this threshold, , is considered faulty, and we follow the next recommended action for that component type.
Success-rate Inference
The second inference strategy follows the approach of offline RL and causal inference methods by combining reward with propensity [33, 59] to ensure that we allow policy improvement while avoiding an extrapolation penalty. For rare actions for a given scenario (i.e., off-policy actions), the model’s estimate of may be unreasonably high [17]. This is further exacerbated by occasional noise in the data (e.g., mis-recorded actions).
In a physical datacenter, prescribing an action with zero historical propensity (an out-of-distribution action) could result in unnecessary actions, workmanship issues, loss of trust by technicians, or possibly even cause hardware damage (e.g. swapping high-voltage components while not powered down). We closely follow the work of Kumar et al. [33] in using propensity as a threshold for selecting a support-set of eligible actions. This enables the policy to diverge from and improve upon the historic distribution while avoiding unreasonable actions (Appendix § 15).
Expert Label Collection
Predicting component-level faults currently necessitates ground-truth binary labels at the Field Replaceable Unit (FRU) level. In practice, such labels are seldom derived automatically from datacenter operational data, as the inherent noise and ambiguity in telemetry preclude the generation of high-fidelity ground truth. Supervised learning in this domain requires domain experts to retroactively audit these sequences to infer the underlying root cause, a process prone to noise and ambiguity. For example, if multiple components are reseated simultaneously to resolve a single failure, attributing the fault to a specific FRU becomes non-trivial.
To eliminate the "human-in-the-loop" bottleneck and accelerate development velocity, our system optimizes for repair action prediction rather than explicit root-cause labeling. By leveraging repair trajectories directly in the training objective, we mitigate the need for manual annotation while capturing the many-to-one mapping between actions and hardware recovery. However, to ensure architectural integrity, we still evaluate the model against a curated "golden set" of recent, expert-filtered repair trajectories that represent the most prevalent hardware syndromes observed in production.
Experiments
Our experimental evaluation is designed to answer three primary questions regarding the efficacy of learning-based diagnostic systems in large-scale data centers: (1) Real-World Impact – Does a learning-based system outperform meticulously maintained, rule-based expert systems in a live production environment? (2) Sample Efficiency and Transfer Learning – Can a foundational model
| Hardware Platform | Mean TTR Reduction | QA Delivery Reduction | Automated Diagnostic % (improvement) |
| H100 | 18.3% ± 7.6% | 28.3% ± 10.7% | 93.9% (+10.7%) |
| TPU | 16.1% ± 2.6% | 25.1% ± 2.5% | 82.8% (+9.9%) |
caption: A/B Production Results: Repair Time Improvements and Automation Coverage. ± denotes one standard error (Appendix § 13).
:::
approach significantly reduce the time and data required to support newly introduced hardware platforms? (3) Multi-Platform Generalization – How does a single, unified foundational model compare to bespoke, single-platform models across diverse GPU and TPU architectures?
Real-World Production Impact (A/B Testing)
To evaluate the operational impact of HARDWAREDX, we conducted a live A/B test in a production environment, comparing its performance against a baseline of the deployed production rules system, which represents a substantial, ongoing engineering investment. Time To Repair (TTR), or downtime, refers to the mean time a machine spends in the test-diagnose-repair loop before returning to serving, excluding install time. Quality Assurance (QA) delivery measures the repair-case percentile needed to reach the contractual cluster-turnover SLO. This rules-based system is the production system that preceded HARDWAREDX: many hundreds of engineer-maintained rules add “badness” scores to the failure modes of each component, and the system prescribes the repair strategy for the highest-scoring component failure mode (Appendix § 12).
We deployed HARDWAREDX to predict the faulty components (Task 1) and prescribe initial repair actions across two major hardware categories: the established H100 platform and a newly introduced TPU platform. The A/B analysis is based on 225 H100 repair windows and 2,567 repair windows on the TPU platform. Traffic is randomized at the machine level: 20% of TPU machines and 50% of H100 machines (a lower-volume pilot) are served by HARDWAREDX, and the remainder by the rules-based system. Appendix § 13 gives further details, including how standard errors are computed.
As shown in Table 1 and Figure 3, HARDWAREDX yielded substantial improvements across all key operational metrics. For the H100 fleet, HARDWAREDX reduced the mean Time To Repair (TTR) by 18.3% and reduced quality-assurance (QA) delivery by 28.3%. For the TPU platform we observed similar reductions of 16.1% for mean repair time and a 25.1% improvement for QA delivery. The large improvement on QA delivery demonstrates the model’s ability to handle more complex cases with comorbid failures that typically confound rule-based heuristics and require time-consuming manual diagnosis. Appendix § 6 presents two such failure patterns, multiple missing GPUs and comorbid DIMM and PCIe failures, where HARDWAREDX improves over the rules-based system. These long-tail cases have an outsized impact on deployment time for new datacenter rollouts due to strict QA standards. Finally, the overall automated diagnostic coverage improved by 10.7% on H100 (83.1% to 93.9%) and 9.9% on the TPU platform (72.9% to 82.8%), proving that ML methods can address a broader distribution of failure modes.
Figure 3. Repair time improvements from deploying HARDWAREDx on a high-volume TPU platform. HARDWAREDx improves repair times across the distribution.
Transfer Learning and Sample Efficiency
A critical bottleneck in data center operations is the "cold-start" problem: when a new compute platform is introduced, it lacks the historical failure data necessary to train a robust diagnostic model from scratch. We evaluated HARDWAREDX’s ability to leverage foundational knowledge to accelerate this adaptation.
Figure 4 demonstrates the sample efficiency and accelerated deployment of our transfer learning approach. We compared a "cold-start" model (trained from scratch on H200 data) against the HARDWAREDX foundation model, which is pre-trained on H100 telemetry and incrementally fine-tuned. HARDWAREDX provides critical "Day Zero" utility by utilizing virtualized graph representations (§ 3) and LLM-based embeddings to recognize failure patterns in new topologies before platform-specific labels exist. Consequently, the model achieves production-grade performance with 10x less data than the cold-start baseline. After 6 months of development, the number of unique observed failure modes was still only half as much as those available on prior platforms. The ability to learn from prior platforms halved the onboarding effort: HardwareDx reached production readiness prior to hardware GA, a significant improvement over the previous generation’s four calendar months post-GA requirement. Table 2 shows how HardwareDx pulls forward the model readiness relative to hardware GA as measured by job-level accuracy and recall. All models in Table 2 are re-evaluated on the same unified held-out set. Historical versions of the rules-based system cannot be replayed, so the Rules@HW-GA row uses the recommendations the production rules made at the time of each repair. Because the rules are continuously updated by a large engineering team, this baseline already includes later rule enhancements, which biases the comparison in favor of the rules (Appendix § 12.§ 4).
Figure 4. Effect of pretraining on transfer learning efficiency. Foundation model pretraining enables onboarding of new platforms with less training data volume.
| Job Acc. | Job Recall | |
| Rules @HW-GA | 0.596 | 0.722 |
| HardwareDx-v1 @HW-GA | 0.521 | 0.664 |
| HardwareDx @HW-GA | 0.722 | 0.776 |
| HardwareDx-v1 @HW-GA+4mo | 0.749 | 0.819 |
| HardwareDx @HW-GA+4mo | 0.942 | 0.969 |
Table 2. Performance development relative to general availability of H200 hardware (HW-GA). HardwareDx-v1 is the prior generation of HardwareDx models (no pretraining).
Foundation Model Multi-Platform Performance
Next, we evaluate the offline performance of our multi-task foundation model on the component-level fault prediction task. Figure 5 compares the performance of a foundation model trained on all related platforms vs a model trained on all available data for a single platform.

Figure 5. F1-score performance comparison of the foundation model vs single platform models. Foundation model performance is generally similar or slightly better than the single platform variants.
The results demonstrate that the unified foundational model is highly competitive with, and often superior to, specialized single-platform models. For instance, on the high-volume TPU v7x platform, the foundation model achieves an F1 score of 0.9892, narrowly beating the single model’s 0.9877. On newer platforms where data is less ubiquitous, the foundation model’s learned structural priors provide a distinct regularization advantage, maintaining high recall without sacrificing precision.
Ablations on GNN Architecture
To justify the architectural design of HARDWAREDX, particularly the hybrid graph representation discussed in § 3, we evaluate the impact of different GNN layers and the inclusion of the Graph Transformer layer. As shown in Table 3, the inclusion of the global attention mechanism (Transformer layer) is critical for handling the long-range dependencies and disconnected software nodes inherent to our diagnostic graphs, consistently improving F1 scores, accuracy, and recall. Furthermore, among the local message-passing layers, GATv2 [7] significantly outperforms standard GIN [76], validating our choice to use attention-based routing for local topology. An R-GCN [61], which keeps the heterogeneous graph and learns relation-specific weights, does not outperform GATv2 on the virtualized graph either alone (0.974 vs. 0.985 F1) or with the Transformer layer (0.988 vs. 0.990 F1), so virtualization slightly improves accuracy.
| Architecture | F1 Score | Accuracy | Recall |
| GATV2+Transformer | 0.990 | 0.988 | 0.991 |
| GIN+Transformer | 0.987 | 0.977 | 0.980 |
| R-GCN+Transformer† | 0.988 | 0.980 | 0.985 |
| GATV2 | 0.985 | 0.978 | 0.986 |
| GIN | 0.943 | 0.924 | 0.932 |
| R-GCN† | 0.974 | 0.953 | 0.962 |
| Transformer-Only (w/ pos & path) | 0.929 | 0.846 | 0.848 |
| Shared Prediction Head | 0.979 | 0.979 | 0.988 |
| No LLM Features | 0.444 | 0.304 | 0.314 |
| GBDT (flattened features)‡ | 0.540 | 0.286 | 0.293 |
| MLP (flattened features)‡ | 0.514 | 0.283 | 0.294 |
Table 3. Ablation on different architecture choices using TPU v7x for evaluation. Metrics are component-level F1, job-level Accuracy, and job-level Recall. †R-GCN operates on the heterogeneous graph without virtualization and learns relation-specific weights. ‡GBDT and MLP are non-GNN baselines.
Table 3 also demonstrates the necessity of structural representation and LLM encodings. Removing the LLM features creates massive performance degradation. Removing the GNN layers and using a Transformer-Only architecture results in a fairly large performance degradation. Note that the Transformer-only model requires positional and device-path encodings to function, which the full GNN model deliberately ignores. We use GNNs because they explicitly provide permutation equivariance and weight sharing across structurally equivalent subgraphs, such as parallel PCIe chains. This allows the model to structurally disambiguate components (e.g., distinguishing whether two failing GPUs reside on the same or different CPU/PCIe chains) without needing additional training examples to transfer a learned pattern from one chain to another. We also analyze a few non-GNN models. We implement Gradient Boosted Decision Trees (GBDT) and a simple Multi-Layer Perceptron (MLP) model using detailed component-level features and aggregated machine-level features. Without the machine topology information, performance significantly degrades (0.54 F1 for GBDT, 0.51 for MLP). Finally, using a shared prediction head across all components slightly degrades performance compared to utilizing individual prediction heads per component.
Conclusion
In this paper, we present HARDWAREDX, a deployed, foundation model system designed to automate hardware diagnostics and prescribe repair actions in large-scale AI datacenters. By shifting away from brittle, manually curated rule-based heuristics, HARDWAREDX directly addresses the escalating complexity and comorbidity of hardware failures in modern heterogeneous computing fleets.
Our approach demonstrates the practical viability of applying representation learning to physical infrastructure operations through several key design choices such as graph representation of machine topology, universal graph format, and multi-platform foundation model training. Crucially, HARDWAREDX effectively mitigates the “cold-start” data bottleneck inherent to new hardware deployments. Our live production A/B tests demonstrate substantial operational improvements, including reducing mean time-to-repair by up to 18.3% and successfully reducing diagnostic QA for new builds by up to 28%. Furthermore, offline evaluations confirm that our transfer learning approach achieves competitive diagnostic accuracy on new platforms up to multiple months earlier than cold-start models.
Ultimately, HARDWAREDX highlights the immense potential of adapting foundational ML techniques for systems operations. By transitioning from reactive, human-maintained rulesets to proactive, data-efficient models, we pave the way for more resilient, self-healing datacenter ecosystems capable of sustaining the next generation of large-scale AI workloads.
Case Studies
We present two case studies on failure patterns where HARDWAREDX improved performance over the rules-based system. These failures showcase two primary advantages of GNN-based representation learning: topology awareness and comorbidity handling. Additionally, the learned patterns impact multiple platforms, highlighting how the foundation model accelerates development velocity for new hardware iterations.
Case 1: Multiple GPUs Missing
This case study highlights the advantage of topology awareness through GNNs. It concerns machines reporting multiple GPUs as missing or unreachable. While a single missing GPU is usually a straightforward indicator of a faulty GPU, cases involving multiple missing GPUs are more complex and challenging for the rules-based system.
Our findings demonstrate high interdependence on the structural topology around the afflicted GPUs (Figure 6). When two GPUs are missing, two distinct scenarios exist based on the PCIe chains. If the same PCIe chain controls both missing GPUs, the model correctly predicts that the PCIe chain requires repair. Conversely, if different PCIe chains control the GPUs, the model learns that GPU repairs are necessary, indicating that either the GPUs or the backplane is faulty. When more than two GPUs are missing, the model typically infers that the backplane is faulty or software issues are present. Table 4 presents the breakdown of recommended repairs.

Figure 6. Illustration of topology-aware diagnosis in Case 1 (Multiple GPUs Missing). Identical Missing telemetry signals on GPUs lead HARDWAREDX to distinct repair predictions based on the underlying PCIe hierarchy: (a) two missing GPUs sharing a PCIe chain implicate the shared PCIe chain (76.3%), (b) two missing GPUs across separate PCIe chains localize the fault to the individual GPUs (8.5%), and (c) more than two missing GPUs across chains escalate to the GPU backplane (6.8%) or software (8.5%).
Because this diagnosis requires attention to the machine’s structural layout, it is difficult for rules-based systems to detect these patterns. However, HARDWAREDX naturally learns to distinguish between them.
Case 2: DIMM Errors with PCIe Low Throughput
HARDWAREDX excels at handling comorbid failures. This case examines a scenario involving both excessive DIMM errors and low PCIe throughput. Through HARDWAREDX, we determined that the DIMM was the ultimate root cause of the issue.
Individually, the rules handle these failures gracefully: excessive DIMM errors trigger DIMM repairs, and low PCIe throughput triggers PCIe chain repairs. However, when both modes appear, the rules trigger both responses. This leads to either extraneous part swaps or increased repair iterations if PCIe repairs are prioritized over the DIMM.
| Category | Percentage |
| GPU Backplane | 6.8% |
| GPU repairs | 8.5% |
| PCIe repairs | 76.3% |
| Software | 8.5% |
caption: tbl:4: Breakdown of recommendations by HARDWAREDX when encountering multiple missing GPUs. Cases are categorized by the most involved repair recommended (Backplane > GPU > PCIe > Software). Cases with comorbid symptoms outside these categories (9.7%) were excluded.
:::
In this scenario, a definitive correct answer exists. The model learns to prioritize DIMM repairs when these symptoms are comorbid. Further analysis revealed that faulty DIMMs with reduced capacities were throttling bandwidth, thereby causing the PCIe errors. This learned pattern facilitated additional diagnostic development and rule adjustments, affecting 2.9% of all PCIe low-throughput cases on ML platforms.
Related Work
Log Analysis and Diagnostics: Existing literature on general diagnostics addresses log analysis and anomaly detection through various methodologies [9, 27, 41]. Initial approaches utilized statistical extraction techniques to process system logs [40]. Subsequent research introduced natural language processing methods, including self-supervised training for anomaly detection (LogBERT, [23]) and prompt-based few-shot learning (LogPPT, [34]). Recent studies have evaluated the application of large language models (LLMs) for log-based fault diagnosis. For instance, Sun et al. [65] applied fine-tuned LLMs alongside fault-oriented log summaries to process extended log sequences. In the context of distributed training, Jiang et al. [29] analyzed spatial and temporal log patterns to diagnose failures in large-scale environments. Additionally, the representation of unstructured telemetry data has been explored using distilled text embeddings, such as those evaluated by Lee et al. [35]. In the context of data center reliability, Li et al. [38] developed an automated intelligent healing framework that suggests software remedies or hardware repair required, but does not suggest the specific hardware repairs.
Graph Neural Networks for Diagnostics: The structural dependencies within physical topologies have been modeled using graph neural networks (GNNs) [31]. To address complex fault propagation, various specialized GNN architectures have been evaluated. For instance, Yang et al. [79] integrated a GNN variant alongside deep reinforcement learning and large language models for self-healing mechanisms in cloud AI software. Similar to Li et al. [38], their work focused on software remedies rather than actual hardware repairs. In edge computing contexts, Su et al. [64] proposed a spatio-temporal graph propagation model (ST-GraphRCA) to trace root causes by analyzing anomaly flows across connected nodes. Outside of the diagnostic regimes, other works have focused on GNNs for chip design [46].
Prescriptive Maintenance and Transfer Learning: Research on maintenance scheduling includes the development of prescriptive maintenance frameworks [20]. For example, Eldele et al. [16] use time-series foundation models to predict bearing failures in rotating machinery. Within this domain, offline reinforcement learning (RL) has been applied to historical datasets to optimize maintenance policies [30, 36]. To manage out-of-distribution actions, several methods incorporate propensity scoring and behavioral cloning to constrain the action space [17, 33, 59].
Graph Foundation Models (GFMs): Graph representation learning has been experiencing a "foundation model" shift away from task-specific training toward models that generalize across diverse graph structures. This field has seen significant recent activity [73, 74].
Early works like DeepWalk [51] and Node2Vec [22] laid the groundwork for learning structural graph embeddings that can be applied to new tasks. As the field turned towards GNNs [31], inductive reasoning became a higher priority with works like GraphSAGE [24] focusing on generalization to new graph structures. Work developed further on generalizing graphs to new nodes [10, 14, 44, 68], new relations, and eventually entirely new graph schemas [18, 72].
In the modern era, additional work has looked more specifically at using language models with graphs to create foundation models [11]. Examples include One-for-All [42] and UniGraph [26], which use LLMs to allow GNNs to bridge different graph domains and schemas. GraphToken [52] reverses this paradigm, using a GNN to encode graphs as tokens for LLM to process. HARDWAREDX incorporates language models for encoding features which supports learning on new symptoms and components.
Historical Context of Repair Diagnosis
Early on, hardware debugging at Google was a manual process without standardized directives, in which engineers and technicians decided which actions to take. As the fleet scaled, technician actions were standardized and simple automated rules were developed. These systems were long considered adequate, as individual components were inexpensive, diagnostic signals were more straightforward, and falling back to manual diagnosis was not too costly when downtime was less critical. With AI datacenters this is no longer the case. Components are very expensive, the deeper connectivity of machines leads to more complicated failures, and repair time is increasingly costly. The rules have evolved into a more complex score-based system (Appendix § 12), but it has not kept pace with the rate of new platform introductions. Our early ML experiments with standard approaches, such as featurized MLPs and decision forests, did not reach the performance necessary to improve on the rules-based system, which led to the GNN-based approach presented here.
Limitations and Future Work
Reproducibility and Data Constraints: Direct replication of our empirical results is constrained by the proprietary nature of datacenter hardware telemetry, fault logs, and maintenance records, which cannot be publicly released due to security and confidentiality requirements. Additionally, live production A/B tests cannot be replayed historically. To maximize conceptual and algorithmic reproducibility, we provide comprehensive architectural specifications (§ 3), formal graph schemas and edge categories (Appendix § 10), feature node keys and component types (Appendix § 11), mathematical formulations of inference heads (Appendix § 15), and extensive ablation studies across multiple architectures and baselines (Table 3). These details are intended to enable researchers and infrastructure operators to implement graph-based diagnostic models within their own compute environments.
Baseline Comparison Limitations: Because production rule engines cannot be backtested against historical machine states without active re-execution, our longitudinal comparison in Table 2 evaluates the production rules available at HW-GA rather than a retrospectively modified Rules@HW-GA+4mo baseline, as discussed in Appendix § 12.
Data Limitations: Our formulation for task 3 suffers from issues of temporal credit assignment [53, 66]. Specifically, our data formulation assigns a success label of 0 to all intermediate steps in a repair trajectory, assigning 1 only to the final passing step. This risks penalizing necessary partial repairs in co-morbid failure scenarios. Furthermore, Equation 4 evaluates the global test outcome against a binary mask of actions, assigning the exact same success signal to all concurrent actions (e.g., a necessary action on component A and an unnecessary action on component B), which introduces causal confounding. This can further obfuscate interpretation as looking at the joint predictions of two correlated actions can imply a different interpretation than looking at the joint predictions for two independent actions.
Future work will incorporate advances in Credit Assignment to better assign labels (rewards) to specific actions [2, 56].
Latency: Serving ML models such as ours introduces additional services and computation, leading to higher latency. However, given that testing can take many hours and wait-times for technicians are even higher, the diagnosis latency is not a consideration for our system.
Scope of Foundation Model and Trajectory Supervision: In this work, "foundation model" refers to a domain-specific graph architecture that pretrains across heterogeneous datacenter machine topologies to learn shared structural priors. While this enables effective cross-platform transfer within internal fleets, HARDWAREDX is not an open general-domain model and does not operate in a fully self-supervised manner. Downstream fine-tuning and deployment still depend on platform-specific topology definitions, telemetry schemas, and curated technician repair trajectories. Evaluating transferability on open or synthetic infrastructure benchmarks without proprietary telemetry remains future work.
Fully-Unsupervised Systems: While HARDWAREDX greatly improves efficiency via foundation model and transfer learning, there is still the requirement for some amount of labeled data and manual effort. This is a limitation we are actively addressing. Appendix § 15 presents a label-free inference approach and Appendix § 16 presents initial results. Future work will further develop our ability to transition to fully unsupervised foundation model systems.
Graph Construction
Heterogeneous graph. We use standard approaches to encoding heterogeneous graphs. A heterogeneous graph is defined by a collection of node sets and a collection of edge sets, where each edge set has an associated source node set and target node set.
Node sets. Node sets include different hardware component types (e.g., cpu, gpu, pcie_cable), different telemetry readings (e.g., sensor, bandwidth), errors (e.g., dimm_errors, oob_errors), and diagnostic symptoms (symptoms). Appendix § 11 lists the feature node types and component types in full.
Edge sets. Edge sets fall into three categories. Physical edge sets connect different component types (e.g., cpu-to-pcie_cable represents a PCIe cable plugged into a CPU connector). Internal edge sets capture physical or logical connections within a component (e.g., the cpu-to-cpu edge set links a CPU’s PCIe port subcomponent to the specific CPU node it belongs to). Feature edge sets connect feature nodes to the component that generated them (e.g., sensor-to-cpu and symptom-to-gpu).
Graph scope. Each machine is represented as a single, self-contained graph. We do not connect different machines in the graph, as hardware topologies and physical faults are isolated at the individual chassis/tray boundary. This keeps the graph size small and inference efficient. Applying a similar approach to inter-node communication hardware is a possible extension of this work. Each platform generally has a single physical topology, though there can occasionally be more than one hardware variant for the same platform. The feature nodes (telemetry, symptoms, and errors) vary from graph to graph in both structure and value.
Homogenization. To homogenize the heterogeneous inputs (telemetry floats, categorical symptoms, text embeddings), each node type is mapped to a shared -dimensional latent space using type-specific projection heads: MLPs for continuous values and lookup tables for discrete tokens. This aligns disparate feature dimensions before the shared GNN-Transformer layers. The homogenization process creates a mirror graph in which all nodes are converted to a single node set and all edges to a single edge set, mirroring the heterogeneous topology.
Input Features and Component Types
Feature Nodes
Each feature is dynamically connected to the component that generates it in the graph topology; features not generated from a component are connected to the “software” node. Each feature becomes a node in the graph with multiple feature values associated with it. For instance, a “DIMM Error” feature node has a string feature for the error, a float feature for the maximum error rate, and a float feature for the configured threshold. Table 5 lists the feature node types. Not every feature is used on every platform, and a small number of features are omitted for IP reasons.
| Feature Node | Feature Keys |
| DIMM Error | Error (message / error type); Max Rate (max errors per minute); Threshold (configured error-rate threshold) |
| CPU Error | Error (message / error type); Max Rate (max errors per minute); Threshold (configured error-rate threshold) |
| Sensor | Name (sensor identifier); Unit (degrees, volts, RPM, watts, etc.); State (health state: GOOD, DETECTED, MISSING, HOT, COLD, etc.); Violations (number of out-of-range readings) |
| Component Bandwidth | Flow (connection type: GPU-to-GPU or GPU-to-CPU); Bandwidth (measured value); Threshold |
| Compute Performance | TFlops; Precision (e.g., FP16, FP32, INT8); Threshold (TFlops threshold) |
| Symptom | Name (symptom identifier); Test Name (test that produced the symptom); Pass/Fail Status; Message (detailed error or diagnostic output) |
| Bus Connection Error | Expected / Actual (string reading); Expected / Actual Width; Expected / Actual Speed; Expected / Actual Enabled; Expected / Actual PCIe Type; Expected / Actual Lanes In Use |
| PCIe Error | Error Type; Max Errors Per Minute; Avg Errors Per Minute; Threshold |
| In-Band GPU XID Error | Count; PCIe (devpath / endpoint identifier); Link; Fatal (boolean); Error Code (XID numeric code); Error Type; Parallel Test (concurrent test running during error) |
| Out-of-Band GPU XID Error | Count; Link; Fatal; Error Code (XID numeric code); Error Type |
| GPU Monitor / BMC Error | Measurement Name; Message Arg (argument extracted from raw Redfish/BMC log); Message (error text) |
| GPU Count Verification | Count (detected GPUs); Expected Count; Abnormality Degree (difference between expected and actual) |
| Field Diagnostics | Name; Type; Value (raw stdout output or diagnostic string) |
| Prior Maintenance Action | Action Type; Preceding Test (test preceding the repair action); Action ID |
| Prior Historical Symptom | Name; Test Name; Pass/Fail Status; Message |
| Granular Component Type | Component Type (more detailed component type) |
Table 5. Feature node types and their feature keys.
Symptoms are the most important feature (Table 6) and are emitted as structured results by all diagnostic tests. The data sources for the features are:
| Ablated Feature Node Type | Component F1 Score | Job Accuracy | Job Recall |
| Full Model (Reference) | 0.9715 | 0.9625 | 0.9764 |
| symptom | 0.3532 | 0.2426 | 0.2443 |
| pcie error | 0.9708 | 0.9599 | 0.9695 |
| sensor | 0.9720 | 0.9634 | 0.9686 |
| dimm error | 0.9723 | 0.9642 | 0.9808 |
| bus error | 0.9747 | 0.9625 | 0.9764 |
| cpu error | 0.9754 | 0.9599 | 0.9808 |
Table 6. Leave-one-out feature ablation on TPU v7x at 10k training steps, ordered from lowest to highest F1 without the feature.
All diagnostic tests: Symptom.
Ongoing error monitors: DIMM Error, CPU Error, PCIe Error, Bus Connection Error.
Ongoing sensor monitors: Sensor.
Specific telemetry tests: Component Bandwidth, Compute Performance.
Kernel logs: In-Band GPU XID Error.
BMC/Redfish logs: Out-of-Band GPU XID Error, GPU Monitor / BMC Error.
NVIDIA-specific diagnostic tests: GPU Count Verification.
Standard output of field diagnostics: Field Diagnostics.
Internal databases: Prior Maintenance Action, Prior Historical Symptom.
Component Nodes
The component node types are: Accelerator, Accelerator Tray, CPU, CSSD, NIC, DIMM, Fabric Tray, Fan, ICI Cable, Interposer, Motherboard, NVSwitch, PCIe Cable, PCIe Adapter Module, Power Cable, Power Supply, Software, SSD, and Thermal Sensor. Components carry the label information (FRU status, faulty, actions performed, action success). Each component also has a device path, but this feature is deliberately ignored to force the model to learn over the topology structure. The initial representations of component nodes are categorical embeddings of the component type. All sub-FRU components are categorized by the component type of their FRU, but may have different “Granular Component Type” features.
Feature Importance Ablations
To further validate the features, we analyzed feature importance on both the TPU v7x and H100 platforms. On TPU v7x, we performed a leave-one-out ablation across the six most common feature node types (Table 6), where lower F1 score, job accuracy, and job recall when a feature is removed indicate higher importance. As expected, symptom features are by far the most critical. These features are derived from hardware diagnostic tests designed by engineers to test the ML accelerators and accompanying hardware. Without them, most failures on TPU v7x cannot be caught, whereas removing any single one of the other five feature types produces metrics within the noise floor of the full model.
On H100, which has 13 telemetry and diagnostic feature node types, we first evaluate the standalone predictive power of each feature type by training single-feature models that keep only one feature node type at a time and remove the other 12 (Table 7, mean one standard error over three seeds). Keeping only symptom matches the full 13-feature model at convergence (0.9697 0.0013 vs. 0.9691 0.0006 F1). Among non-symptom signals, broad diagnostic and telemetry node types carry substantial standalone signal, led by bus_error (0.5725 F1), field_diag (0.5531 F1 and 0.6686 job accuracy), sensor (0.4654 F1 and 0.5892 job accuracy), gpu_count (0.3506 F1), and bandwidth (0.1780 F1). Subsystem-specific error counters (dimm_error, xid_error, oob_xid_error, tflops, cpu_error, pcie_error, and gpu_error) only fire on machines experiencing those specific fault classes and thus yield low overall recall when used in isolation.
| Kept Feature Node Type(s) | Component F1 Score | Job Accuracy | Job Recall |
| Full model (all 13 features) | 0.9691 0.0006 | 0.9688 0.0002 | 0.9783 0.0006 |
| All except symptom (12 features) | 0.9270 0.0060 | 0.9075 0.0034 | 0.9151 0.0037 |
| only symptom | 0.9697 0.0013 | 0.9707 0.0007 | 0.9791 0.0006 |
| only bus error | 0.5725 0.0003 | 0.5031 0.0002 | 0.5053 0.0002 |
| only field diag | 0.5531 0.0030 | 0.6686 0.0017 | 0.6740 0.0016 |
| only sensor | 0.4654 0.0012 | 0.5892 0.0003 | 0.5909 0.0009 |
| only gpu count | 0.3506 0.0006 | 0.4940 0.0003 | 0.4954 0.0005 |
| only bandwidth | 0.1780 0.0895 | 0.3850 0.0179 | 0.4062 0.0186 |
| only dimm error | 0.0595 0.0000 | 0.3926 0.0000 | 0.3926 0.0000 |
| only xid error | 0.0563 0.0021 | 0.3963 0.0009 | 0.3990 0.0009 |
| only oob xid error | 0.0094 0.0009 | 0.3737 0.0006 | 0.3739 0.0005 |
| only tflops | 0.0085 0.0000 | 0.3732 0.0002 | 0.3734 0.0000 |
| only cpu error | 0.0009 0.0009 | 0.3688 0.0002 | 0.3690 0.0000 |
| only pcie error | 0.0000 0.0000 | 0.3690 0.0000 | 0.3690 0.0000 |
| only gpu error | 0.0000 0.0000 | 0.3690 0.0000 | 0.3690 0.0000 |
Table 7. Single-feature inclusion (“keep-only”) ablation on H100 (mean one standard error across 3 seeds at 30k steps). The top two reference rows show the full 13-feature model and the 12-feature model with symptom removed. Each remaining row keeps only the listed feature node type and removes the other 12, ordered by descending F1.
We also performed a leave-one-out ablation across all 13 feature types on H100. Removing symptom reduces component F1 from 0.9691 to 0.9270 ( percentage points) and job accuracy from 0.9688 to 0.9075 ( percentage points). Unlike TPU v7x, where removing symptom reduces F1 to 0.3532, combining the 12 non-symptom telemetry, counter, and diagnostic feature types on H100 still recovers over 92% component F1 and 90% job accuracy. Removing pcie_error produces a small dip to 0.9663 F1 (0.9648 job accuracy), while removing any single one of the remaining feature types leaves F1 within 0.9676–0.9699 and job accuracy within 0.9670–0.9715, confirming that overlapping telemetry and symptom signals compensate when any individual non-symptom feature source is dropped.
Rules-Based Diagnostic System
The rules-based diagnosis system is the automated diagnostic system used to maintain hardware health prior to HARDWAREDX, and it is the baseline in our production A/B tests (§ 4). It operates primarily on a badness model, in which components accumulate or lose “badness” (a value between 0 and 1) based on incoming signals. As a simple example, if a GPU temperature reading exceeds a threshold, a rule may add a badness of to the GPU’s “thermal” failure mode. The production system contains many hundreds of rules of varying complexity.
Diagnosis Process
Signal standardization. The required diagnostic signals are first pulled from health databases and standardized into signals consisting of a reporter, a symptom name, an affected device path, and a timestamp.
Rule execution. The system checks the registry of all rules and activates any that are triggered by the present symptoms. Each FRU component has a number of possible failure modalities (e.g., connectivity, thermal, power). Triggered rules add a badness delta to specific dimensions of one or more components (e.g., increasing the badness score for thermal issues on the CPU). Rules can also have an “else” condition, under which a passing test adds a negative badness delta to certain failure modes. More complicated rules can be constructed using compound if-conditions. The badness scores of all triggered rules are aggregated by part-dimension; when multiple rules trigger at the same time (comorbidities), this aggregation determines which failure mode takes priority.
Repair selection. The system identifies the part-dimension pairs with the highest badness score. Based on that failure mode, it looks up a corresponding repair strategy and outputs a sequence of actions. These are compared against prior actions, with post-processing to avoid repeating actions and to improve efficacy.
Types of Rules
There are three general classes of rules that can be added to the rules registry:
Standard rules: If the reporter (e.g., a PCIe connectivity test) and symptom (e.g., low throughput) match, the rule applies the specified badness deltas.
Rate-limited rules: Identical to standard rules, but triggered at most once per device path per iteration. This is useful for repetitive reporters or multiple test loops, preventing artificial badness inflation.
High-confidence overrides: For highly unambiguous signals, the rule directly triggers a specific repair strategy.
In addition to these configurable rules, certain rules are written directly in code to handle more complex logic.
Development and Maintenance of Rules
Every rule is onboarded via direct engineering effort. Because the rules are mostly config-based, engineers from new-platform onboarding teams as well as from external teams and vendors are able to onboard rules effectively. While updating the rules is straightforward, discovering and determining which rules to add is a highly involved process requiring significant time and engineering effort. Badness scores can be initially determined by hardware and diagnostic engineers, but they require ongoing refinement as data is collected and performance gaps are found.
Rules Baseline in Longitudinal Comparisons
Table 2 evaluates the models that had been developed as of HW-GA and as of HW-GA+4mo, re-evaluated on a unified held-out dataset for consistent comparison. Due to the complexity of the rules-based system, there is no way to evaluate historical versions of the system as a whole. Instead, we use the rules’ recommendations at the time of the actual hardware repair as the Rules@HW-GA
| Platform | Unsupervised Dataset Size (Trajectories) | Labeled Cases | Median Graph Size (Nodes) | Unique Labels |
| TPU v5p | 111,764 | 5,236 | 874 | 59 |
| TPU v7x | 43,987 | 7,529 | 992 | 66 |
| TPU v7 | 1,381 | 245 | 1,142 | 29 |
| H100 | 81,015 | 6,780 | 1,613 | 85 |
| H200 | 7,558 | 1,838 | 1,276 | 80 |
| B200 | 18,481 | 1,667 | 1,379 | 91 |
caption: HardwareDx Component Training Dataset Statistics
:::
baseline. Because the rules are continuously updated in production by a large engineering team, this baseline already includes ongoing rule enhancements. This biases the comparison in favor of the rules-based system, making the reported improvements of HARDWAREDX conservative.
Production A/B Evaluation Details
Baseline. The A/B tests compare HARDWAREDX against the deployed production diagnostic system, i.e., the rules-based system described in Appendix G. This system sees heavy, ongoing engineering investment (greater than that of HARDWAREDX) and is a highly competitive baseline.
Assignment. Traffic is split at the machine level between the production rules-based system and HARDWAREDX. We use 20% of machines for the TPU platform and 50% of machines for the H100 pilot, which has lower repair volume. The analysis covers 225 H100 repair windows and 2,567 TPU repair windows.
Metrics. The reduction in downtime (mean TTR) is computed from the mean time a machine spends in the test, diagnosis, and repair loop before returning to serving; install time is excluded. The QA delivery metric is based on contractual service level objectives: to turn over a cluster of machines, a certain percentage of compute nodes must be fully online and functioning. We compute the percentile of repair cases that must be completed to reach that service level objective and report the repair time at that percentile.
Uncertainty. Standard errors of the mean TTR are computed with classical parametric estimation, , where is the sample standard deviation and the number of repair windows. Standard errors for the QA delivery reduction, which is a percentile, are computed non-parametrically using the Woodruff method [75], based on the binomial distribution of order statistics. For the H100 fleet, HARDWAREDX reduced mean TTR by 18.3%±7.6% and QA delivery time by 28.3%±10.7%. For the TPU platform, it reduced mean TTR by 16.1%±2.6% and QA delivery time by 25.1%±2.5% (± one standard error). Even though the H100 analysis is based on fewer repair windows, the improvements remain statistically significant.
Data Collection
For supervised data collection, we utilize expert recommendations. For each job being labeled, the expert will look at the full case history and the repair-ability guidelines for the failing diagnostic tests before selecting which components should be touched. This is a highly technical and involved process. tbl:8 shows the collected dataset volumes.
Dataset characteristics and Topology
tbl:8 provides the primary training datasets utilized by HARDWAREDX. TPU v5p and H100 represent our most data-rich environments, each containing over 111,000 and 81,000 trajectories, respectively. The H100 platform shows the most complex physical topology, requiring a median of 1,613 nodes per machine snapshot to represent all components, subcomponents, and feature signals.
Label Diversity
The B200 and H100 platforms show the greatest variety of unique labels (91 and 85, respectively), which indicates a broad capture of failure modes for these architectures. Meanwhile, the H100 platform has the highest number of labeled windows. TPU v7x similarly maintains a highly robust set of ground truth-examples despite having a smaller overall dataset size compared to TPU v5p and H100.
Success-Rate Inference
As mentioned in the paper, the success-rate inference strategy aims to enable HARDWAREDX to directly predict actions based on unsupervised learning. This strategy aims to take the most likely to succeed actions while avoiding unreasonable actions (those that would never be seen in normal data center operations), while also avoiding taking excessive actions, even if reasonable. Actions are chosen based on (1) the component needing an action applied, (2) the action being reasonable to expect, and (3) the action having higher chance of success than any other action for the component.
No-Action Gating
Let denote the “no-action” operation, and let represent the predicted conditional likelihood of the machine passing the next test if no physical repair is performed on component . The inference procedure follows a three-step decision process:
For each component, we first evaluate whether physical intervention is strictly necessary. If the predicted likelihood of success without intervention exceeds a predefined threshold , the system recommends no action for that component. This prevents unnecessary hardware swaps in the presence of transient faults (e.g., software timeouts that resolve upon reboot).
Propensity Filtering
For each component, , where intervention is required (), we define a candidate set of component-action pairs, , by filtering the action space using the behavioral prior learned in Task 2. We only consider actions whose predicted propensity exceeds a threshold :
This bounds our search space to the manifold of historically reasonable technician behaviors, preventing the model from prescribing out-of-distribution or structurally invalid actions (following Kumar et al. [33]). The mathematical gating mechanism () effectively bounds the model’s policy within the manifold of physically safe, historically validated technician behaviors, prioritizing hardware safety over aggressive exploration.
Maximum Likelihood Selection
Finally, from the filtered candidate set , we select the action for component that maximize the conditional likelihood of success, as predicted by Task 3:
If is empty, we fallback to .
By chaining these heads, HARDWAREDX balances aggressive automated repairs with historical safety priors, ensuring that the prescribed actions are both highly likely to succeed and operationally sound.
Simulated Evaluation of Success-Rate Inference
We conducted additional evaluation on the Success-Rate Inference strategy for the unsupervised tasks using a window-based simulation approach. We collect a set of 941 recent repair trajectories of the form
where are the failing tests, is the final passing test, and each is the set of repair actions actually performed at that iteration, excluding "no-action". This is a slight redefinition of notation from what was used in Section 3, but represents the same underlying data. While we cannot know the utility of every action in the trajectory, we can assume that actions are likely to be useful as they led to the final tests passing. To cover a wide range of situations we sample the windows based on the presenting symptoms and the number of iterations it took to fix. This makes the selected windows more varied and more difficult than the natural distribution.
We cannot simulate the effect of running HARDWAREDX’s success-based inference from the start of the window as the observed test results and simulated test results would diverge as soon as the actions diverge. However, we can simulate the effect of running HARDWAREDX independently on each failing test through . Let be the HARDWAREDX suggested sets of actions.
We then evaluate how often HARDWAREDX replicates the final actions in its prior predictions:
We also measure an action-weighted version. This is meant to account for the fact that not every one of the final actions is expected to be necessary. For instance, if there were three parts swapped, potentially only one of those swaps was necessary. This metric gives the model partial credit for replicating a subset of the final actions.
Of the samples that were replicated, we can also measure the estimated iterations it takes to reproduce the final actions.
Table 9 shows the results of this window based evaluation. The HARDWAREDX approach shows a better final action replication rate among these windows while also offering a better estimated improvement in iterations.
| Metric | Existing System | HardwareDx Predict Repair Actions |
| Final Action Replication Rate | 56.23% | 63.42% |
| Action-Weighted Final Action Replication Rate | 62.50% | 73.04% |
| Average Estimated Iterations ( over actual iterations) | -0.29 | -0.50 |
Table 9. Performance Comparison: Existing Automated System vs. HARDWAREDX Predict Repair Actions with Success Rate
Finetuning Two-Platform Model
We run an additional experiment in which we train a two-platform foundation model on H100 and H200 data before fine-tuning on complete B200 data. Table 10 shows these results. Using the foundation model as a starting point improves per component F1-score and job-level accuracy, demonstrating the generalization capability.
| Model | B200 (Finetune Platform) | Job-level Acc | Job-level Rec | Component F1 |
| FM-2Platform-FT | 0.9848 | 0.9939 | 0.9817 | |
| Single | 0.9787 | 0.9939 | 0.9728 |
Table 10. Results of finetuning a foundation model pretrained on H100 and H200 data on the B200 platform.
Additional Ablations
We present additional ablation experiments here. Table 3 in the main paper presented the primary ablation on core GNN choice. Figure 7 shows the results of tuning number of layers. Two layers is clearly not enough to adequately learn the data. While performance does continue to improve, the discrepancy between four and eight layers is much smaller. Interestingly, with only two layers, both GATv2 and GIN perform far worse than with the Transformer included, but with more layers, GATv2 only performs slightly worse while a large gap remains to GIN.
Figure 7. Comprehensive ablation study on the number of layers.
We further present some additional evaluation on calibration metrics for the Task 2 and 3 predictions. We present the Expected Calibration Error and Brier Score of our foundation model in Table 11. Expected Calibration Error is a pure calibration measure: it measures how close the model predicted rate is to the true rate (e.g. of an action being taken or of the next test passing) across different bins [47]. However, this can be gamed by predicting more conservatively, such as only predicting the average rate of incidence. Brier Score measures calibration but also factors in the discriminative power of the predictions [6].
| GNN Layer | Expected Calibration Error (↓) | Brier Score (↓) |
| GATv2 | 0.0015 | 0.0062 |
| GIN | 0.0020 | 0.0067 |
Table 11. GNN Model Calibration on TPU v7x using Expected Calibration Error and Brier Scores (lower is better). The evaluated models both include transformer layers but include different local GNN layers.
Hyperparameters
We conduct hyperparameter sweeps across learning rates of , , , using a cosine decay schedule with 1000 warmup steps. We evaluate per-task batch sizes of [8, 16] (measured in number of graphs per batch). We evaluate latent dimensions of [?, ?]. We evaluate layer depth of [2, 4, 8]. Models are trained with AdamW using and . Foundation model training is done for 50,000 steps (with occasional early stopping). Fine-tuning is performed for 10,000 steps. We split data 80% for training and 20% for evaluation. Prior to production, we confirm the model functions on a small held out set of recent unseen data to provide further validation.
References
- [1]Sami Abu-El-Haija, Bryan Perozzi, Amol Kapoor, Nazanin Alipourfard, Kristina Lerman, Hrayr Harutyunyan, Greg Ver Steeg, and Aram Galstyan. Mixhop: Higher-order graph convolutional architectures via sparsified neighborhood mixing. In international conference on machine learning, pages 21–29. PMLR, 2019.
- [2]Vyacheslav Alipov, Riley Simmons-Edler, N.Yu. Püntintsev, Pavel Kalinin, and Dmitry P. Vetrov. Towards practical credit assignment for deep reinforcement learning. ArXiv, abs/2106.04499, 2021. URL https://api.semanticscholar.org/CorpusID:235368057.arxiv.org/abs/2106.04499
- [3]Uri Alon and Eran Yahav. On the bottleneck of graph neural networks and its practical implications. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=i80P0hC0VH2.arxiv.org/abs/2006.05205
- [4]Paul Borrill. The ghost in the datacenter: Link flapping, topology knowledge failures, and the FITO category mistake. arXiv preprint arXiv:2603.03736, 2026.arxiv.org/abs/2603.03736
- [5]Mirela Madalina Botezatu, Ioana Giurgiu, Jasmina Bogojeska, and Dorothea Wiesmann. Predicting disk replacement towards reliable data centers. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 39–48, 2016.DOI
- [6]Glenn W. Brier. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78:1–3, 1950. URL https://api.semanticscholar.org/CorpusID:122906757.doi.org/10.1175/1520-0493(1950)078<0001:vofeit>2.0.co;2
- [7]Shaked Brody, Uri Alon, and Eran Yahav. How attentive are graph attention networks? In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=F72ximsx7C1.arxiv.org/abs/2105.14491
- [8]Ines Chami, Sami Abu-El-Haija, Bryan Perozzi, Christopher Ré, and Kevin Murphy. Machine learning on graphs: A model and comprehensive taxonomy. Journal of Machine Learning Research, 23(89):1–64, 2022.
- [9]Varun Chandola, Arindam Banerjee, and Vipin Kumar. Anomaly detection: A survey. ACM Computing Surveys, 41:15:1–15:58, 2009. URL https://api.semanticscholar.org/CorpusID:207172599.
- [10]Mingyang Chen, Wen Zhang, Yuxia Geng, Zezhong Xu, Jeff Z. Pan, and Hua zeng Chen. Generalizing to unseen elements: A survey on knowledge extrapolation for knowledge graphs. ArXiv, abs/2302.01859, 2023. URL https://api.semanticscholar.org/CorpusID:256598084.arxiv.org/abs/2302.01859
- [11]Zhikai Chen, Haitao Mao, Jingzhe Liu, Yu Song, Bingheng Li, Wei Jin, Bahare Fatemi, Anton Tsitsulin, Bryan Perozzi, Hui Liu, et al. Text-space graph foundation models: Comprehensive benchmarks and new insights. Advances in Neural Information Processing Systems, 37:7464–7492, 2024.arxiv.org/abs/2406.10727
- [12]Shengkun Cui, Archit Patke, Ziheng Chen, Aditya Ranjan, Hung Nguyen, Phuong Cao, Brett Bode, Gregory Bauer, Saurabh Jha, Chandra Narayanaswami, et al. Characterizing modern GPU resilience and impact in HPC systems: A case study of A100 GPUs. In 2025 55th Annual IEEE/IFIP International Conference on Dependable Systems and Networks Workshops (DSN-W), pages 1–6. IEEE, 2025.DOI
- [13]Anwesha Das, Frank Mueller, Charles Siegel, and Abhinav Vishnu. Desh: deep learning for system health prediction of lead times to failure in hpc. In Proceedings of the 27th international symposium on high-performance parallel and distributed computing, pages 40–51, 2018.DOI
- [14]Daniel Daza, Michael Cochez, and Paul Groth. Inductive entity representations from text via link prediction. Proceedings of the Web Conference 2021, 2020. URL https://api.semanticscholar.org/CorpusID:222177425.DOI
- [15]Vijay Prakash Dwivedi and Xavier Bresson. A generalization of transformer networks to graphs. AAAI Workshop on Deep Learning on Graphs: Methods and Applications, 2021.arxiv.org/abs/2012.09699
- [16]Emadeldeen Eldele, Mohamed Ragab, Xu Qing, Zhenghua Chen, Min Wu, Xiaoli Li, and Jay Lee. Unifault: A fault diagnosis foundation model from bearing data. arXiv preprint arXiv:2504.01373, 2025.arxiv.org/abs/2504.01373
- [17]Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, 2018. URL https://api.semanticscholar.org/CorpusID:54457299.arxiv.org/abs/1812.02900
- [18]Mikhail Galkin, Xinyu Yuan, Hesham Mostafa, Jian Tang, and Zhaocheng Zhu. Towards foundation models for knowledge graph reasoning. ArXiv, abs/2310.04562, 2023. URL https://api.semanticscholar.org/CorpusID:263831485.arxiv.org/abs/2310.04562
- [19]Jiechao Gao, Haoyu Wang, and Haiying Shen. Task failure prediction in cloud data centers using deep learning. IEEE transactions on services computing, 15(3):1411–1422, 2020.
- [20]Alessandro Giacotto, Henrique Costa Marques, and Alberto Martinetti. Prescriptive maintenance: a comprehensive review of current research and future directions. Journal of quality in maintenance engineering, 31(1):129–173, 2025.DOI
- [21]Alessio Gravina, Davide Bacciu, and Claudio Gallicchio. Anti-symmetric DGN: a stable architecture for deep graph networks. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=J3Y7cgZ0OS.arxiv.org/abs/2210.09789
- [22]Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016. URL https://api.semanticscholar.org/CorpusID:207238980.arxiv.org/abs/1607.00653
- [23]Haixuan Guo, Shuhan Yuan, and Xintao Wu. Logbert: Log anomaly detection via bert. In 2021 international joint conference on neural networks (IJCNN), pages 1–8. IEEE, 2021.
- [24]William L. Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. ArXiv, abs/1706.02216, 2017. URL https://api.semanticscholar.org/CorpusID:4755450.arxiv.org/abs/1706.02216
- [25]Tao He, Xue Li, Zhibin Wang, Kun Qian, Jingbo Xu, Wenyuan Yu, and Jingren Zhou. Unicron: Economizing self-healing LLM training at scale. arXiv preprint arXiv:2401.00134, 2023.arxiv.org/abs/2401.00134
- [26]Yufei He, Yuan Sui, Xiaoxin He, and Bryan Hooi. Unigraph: Learning a unified cross-domain foundation model for text-attributed graphs. Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.1, 2024. URL https://api.semanticscholar.org/CorpusID:267770321.arxiv.org/abs/2402.13630
- [27]Junjie Huang, Zhihan Jiang, Jinyang Liu, Yintong Huo, Jiazhen Gu, Zhuangbin Chen, Cong Feng, Hui fen Dong, Zengyin Yang, and Michael R. Lyu. Demystifying and extracting fault-indicating information from logs for failure diagnosis. 2024 IEEE 35th International Symposium on Software Reliability Engineering (ISSRE), pages 511–522, 2024. URL https://api.semanticscholar.org/CorpusID:272770655.arxiv.org/abs/2409.13561
- [28]Junjie Huang, Zhihan Jiang, Zhuangbin Chen, and Michael Lyu. No more labelled examples? an unsupervised log parser with LLMs. Proceedings of the ACM on Software Engineering, 2 (FSE):2406–2429, 2025.DOI
- [29]Zhihan Jiang, Junjie Huang, Guangba Yu, Zhuangbin Chen, Yichen Li, Renyi Zhong, Cong Feng, Yongqiang Yang, Zengyin Yang, and Michael Lyu. L4: Diagnosing large-scale LLM training failures via automated log analysis. In Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering, pages 51–63, 2025.arxiv.org/abs/2503.20263
- [30]Hamed Khorasgani, Haiyan Wang, Chetan Gupta, and Ahmed Farahat. An offline deep reinforcement learning for maintenance decision-making. arXiv preprint arXiv:2109.15050, 2021.arxiv.org/abs/2109.15050
- [31]Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. International Conference on Learning Representations (ICLR), 2017.arxiv.org/abs/1609.02907
- [32]Apostolos Kokolıs, Michael Kuchnik, John Hoffman, Adithya Kumar, Parth Malani, Faye Ma, Zach DeVito, Shubho Sengupta, Kalyan Saladi, and Carole-Jean Wu. Revisiting reliability in large-scale machine learning research. 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA), pages 1259–1274, 2024. URL https://api.semanticscholar.org/CorpusID:273662524.arxiv.org/abs/2410.21680
- [33]Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in neural information processing systems, 32, 2019.arxiv.org/abs/1906.00949
- [34]Van-Hoang Le and Hongyu Zhang. Log parsing with prompt-based few-shot learning. 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 2438–2449, 2023.arxiv.org/abs/2302.07435
- [35]Jinhyuk Lee, Zhuyun Dai, Xiaoqi Ren, Blair Chen, Daniel Cer, Jeremy R Cole, Kai Hui, Michael Boratko, Rajvi Kapadia, Wen Ding, et al. Gecko: Versatile text embeddings distilled from large language models. arXiv preprint arXiv:2403.20327, 2024.arxiv.org/abs/2403.20327
- [36]Sergey Levine, Aviral Kumar, G. Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv, abs/2005.01643, 2020. URL https://api.semanticscholar.org/CorpusID:218486979.arxiv.org/abs/2005.01643
- [37]Yonatan Levitt, Richard Barella, Sam Zeltner, Thomas Musta, Lance Cheney, Gustavo Espinosa, Olivier Franza, and Balazs Gerofi. Fine-grained automated failure management for extreme-scale GPU accelerated systems. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1073–1084, 2025.DOI
- [38]Rui Li, Zhinan Cheng, P. Lee, Pinghui Wang, Yi Qiang, Lin Lan, Cheng He, Jinlong Lu, Mian Wang, and Xinquan Ding. Automated intelligent healing in cloud-scale data centers. 2021 40th International Symposium on Reliable Distributed Systems (SRDS), pages 244–253, 2021. URL http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9603508.DOI
- [39]Yinglung Liang and Yanyong Zhang. Bluegene/l failure analysis and prediction models. In International Conference on Dependable Systems and Networks (DSN’06), pages 425–434. IEEE, 2006.DOI
- [40]Fred Lin, Keyur Muzumdar, Nikolay Pavlovich Laptev, Mihai-Valentin Curelea, Seunghak Lee, and Sriram Sankar. Fast dimensional analysis for root cause investigation in a large-scale service environment. Proceedings of the ACM on Measurement and Analysis of Computing Systems, 4(2):1–23, 2020.
- [41]Qingwei Lin, Hongyu Zhang, Jian-Guang Lou, Yu Zhang, and Xuewei Chen. Log clustering based problem identification for online service systems. 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C), pages 102–111, 2016. URL https://api.semanticscholar.org/CorpusID:10476185.DOI
- [42]Hao Liu, Jiarui Feng, Lecheng Kong, Ningyue Liang, Dacheng Tao, Yixin Chen, and Muhan Zhang. One for all: Towards training one graph model for all classification tasks. ArXiv, abs/2310.00149, 2023. URL https://api.semanticscholar.org/CorpusID:265871676.arxiv.org/abs/2310.00149
- [43]Heting Liu, Zhichao Li, Cheng Tan, Rongqiu Yang, Guohong Cao, Zherui Liu, and Chuanxiong Guo. Predicting gpu failures with high precision under deep learning workloads. In Proceedings of the 16th ACM International Conference on Systems and Storage, pages 124–135, 2023.DOI
- [44]Elan Markowitz, Keshav Balasubramanian, Mehrnoosh Mirtaheri, Murali Annavaram, A. G. Galstyan, and Greg Ver Steeg. Statik: Structure and text for inductive knowledge graph completion. In NAACL-HLT, 2022. URL https://api.semanticscholar.org/CorpusID:250520198.
- [45]Justin Meza, Qiang Wu, Sanjeev Kumar, and Onur Mutlu. Revisiting memory errors in large-scale production data centers: Analysis and modeling of new trends from the field. In 2015 45th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, pages 415–426. IEEE, 2015.DOI
- [46]Azalia Mirhoseini, Anna Goldie, Mustafa Yazgan, Joe Wenjie Jiang, Ebrahim M. Songhori, Shen Wang, Young-Joon Lee, Eric Johnson, Omkar Pathak, Azade Nazi, Jiwoo Pak, Andy Tong, Kavya Srinivasa, Will Hang, Emre Tuncer, Quoc V. Le, James Laudon, Richard Ho, Roger Carpenter, and Jeff Dean. A graph placement methodology for fast chip design. Nature, 594: 207 – 212, 2021. URL https://api.semanticscholar.org/CorpusID:235395490.DOI
- [47]Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. AAAI Conference on Artificial Intelligence, 2015:2901–2907, 2015. URL https://api.semanticscholar.org/CorpusID:6292807.pubmed.ncbi.nlm.nih.gov/25927013
- [48]Daniel Oliveira, Laércio Pilla, Nathan DeBardeleben, Sean Blanchard, Heather Quinn, Israel Koren, Philippe Navaux, and Paolo Rech. Experimental and analytical study of Xeon Phi reliability. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–12, 2017.DOI
- [49]Dylan Patel and Daniel Nishball. 100,000 h100 clusters: Power, network topology, ethernet vs infiniband, reliability, failures, checkpointing. 2024. URL https://www.semianalysis.com/p/100000-h100-clusters-power-network.
- [50]Karthik Pattabiraman, Mihir Patel, and Fred Lin. AIReSim: A discrete event simulator for large-scale AI cluster reliability modeling. arXiv preprint arXiv:2603.07041, 2026.arxiv.org/abs/2603.07041
- [51]Bryan Perozzi, Rami Al-Rfou, and Steven S. Skiena. Deepwalk: online learning of social representations. Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, 2014. URL https://api.semanticscholar.org/CorpusID:3051291.
- [52]Bryan Perozzi, Bahare Fatemi, Dustin Zelle, Anton Tsitsulin, Mehran Kazemi, Rami Al-Rfou, and Jonathan J. Halcrow. Let your graph do the talking: Encoding structured data for LLMs. ArXiv, abs/2402.05862, 2024. URL https://api.semanticscholar.org/CorpusID:267547812.arxiv.org/abs/2402.05862
- [53]Eduardo Pignatelli, Johan Ferret, Matthieu Geist, Thomas Mesnard, Hado van Hasselt, and Laura Toni. A survey of temporal credit assignment in deep reinforcement learning. ArXiv, abs/2312.01072, 2023. URL https://api.semanticscholar.org/CorpusID:265608792.arxiv.org/abs/2312.01072
- [54]Junsong Pu, Yichen Li, Zhuangbin Chen, Jinyang Liu, Zhihan Jiang, Jianjun Chen, Rui Shi, Zibin Zheng, and Tieying Zhang. Errorprism: Reconstructing error propagation paths in cloud service systems. arXiv preprint arXiv:2509.26463, 2025.arxiv.org/abs/2509.26463
- [55]Zhenlian Qi, Junyu Ren, Wensheng Gan, and Philip S Yu. Large language models for fault diagnosis. In 2025 IEEE International Conference on Big Data (BigData), pages 6982–6991. IEEE, 2025.DOI
- [56]Yun Qu, Yuhang Jiang, Boyuan Wang, Yixiu Mao, Qi Cheems Wang, Chang Liu, and Xiangyang Ji. Latent reward: LLM-empowered credit assignment in episodic reinforcement learning. In AAAI Conference on Artificial Intelligence, 2024. URL https://api.semanticscholar.org/CorpusID:274777416.arxiv.org/abs/2412.11120
- [57]Ladislav Rampášek, Michael Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Dominique Beaini. Recipe for a general, powerful, scalable graph transformer. Advances in Neural Information Processing Systems, 35:14501–14515, 2022.arxiv.org/abs/2205.12454
- [58]Bhala Ranganathan, Minghua Ma, Mickey Zhang, Klein Hu, Rakesh Kelkar, and Chetan Bansal. An empirical study of automation gaps in llm serving systems. International Workshop on Cloud Intelligence / AIOps (AIOps ’26), 2026.
- [59]Paul R Rosenbaum and Donald B Rubin. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):41–55, 1983.
- [60]Omkar Salpekar, Rohan Varma, Kenny Yu, Vladimir Ivanov, Yang Wang, A. Sharif, Min Si, Shawn Xu, Feng Tian, Shengbao Zheng, Tristan Rice, Ankush Garg, Shangfu Peng, Shreyas Srivavara, Wenyin Fu, Rodrigo de Castro, Adi Gangidi, A. S. Obratzov, Sharan Narang, Sergey Edunov, Maxim Naumov, Chunqiang Tang, and Mathew Oldham. Training LLMs with fault tolerant HSDP on 100,000 GPUs. ArXiv, abs/2602.00277, 2026. URL https://api.semanticscholar.org/CorpusID:285269506.arxiv.org/abs/2602.00277
- [61]M. Schlichtkrull, Thomas Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. In Extended Semantic Web Conference, 2017. URL https://api.semanticscholar.org/CorpusID:5458500.arxiv.org/abs/1703.06103
- [62]Vilas Sridharan, Jon Stearley, Nathan DeBardeleben, Sean Blanchard, and Sudhanva Gurumurthi. Feng shui of supercomputer memory: Positional effects in DRAM and SRAM faults. In Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, pages 1–11, 2013.DOI
- [63]Vilas Sridharan, Nathan DeBardeleben, Sean Blanchard, Kurt B Ferreira, Jon Stearley, John Shalf, and Sudhanva Gurumurthi. Memory errors in modern systems: The good, the bad, and the ugly. ACM SIGARCH Computer Architecture News, 43(1):297–310, 2015.
- [64]Tianyi Su, Ruibing Mo, Yanyu Gong, and Haifeng Wang. ST-GraphRCA: A root cause analysis model for spatio-temporal graph propagation in IoT edge computing. Sensors, 26(5):1474, 2026.DOI
- [65]Yongqian Sun, Shiyu Ma, Tong Xiao, Yongxin Zhao, Xuhui Cai, Wei Dong, Yue Shen, Yao Zhao, Shenglin Zhang, Jing Han, and Dan Pei. Accurate and interpretable log-based fault diagnosis using large language models. IEEE Transactions on Services Computing, 2025.
- [66]Richard S. Sutton. Temporal credit assignment in reinforcement learning. 1984. URL https://api.semanticscholar.org/CorpusID:60564875.
- [67]Gemini Team et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. ArXiv, abs/2507.06261, 2025. URL https://api.semanticscholar.org/CorpusID:280151524.
- [68]Komal K. Teru, E. Denis, and William L. Hamilton. Inductive relation prediction by subgraph reasoning. In International Conference on Machine Learning, 2019. URL https://api.semanticscholar.org/CorpusID:211082667.
- [69]Jake Topping, Francesco Di Giovanni, Benjamin Paul Chamberlain, Xiaowen Dong, and Michael M. Bronstein. Understanding over-squashing and bottlenecks on graphs via curvature. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=7UmjRGzp-A.arxiv.org/abs/2111.14522
- [70]Kashi Venkatesh Vishwanath and Nachiappan Nagappan. Characterizing cloud computing hardware reliability. In Proceedings of the 1st ACM symposium on Cloud computing, pages 193–204, 2010.DOI
- [71]Borui Wan, Gaohong Liu, Zuquan Song, Jun Wang, Yun Zhang, Guangming Sheng, Shuguang Wang, Houmin Wei, Chenyuan Wang, Weiqiang Lou, et al. Robust LLM training infrastructure at bytedance. In Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles, pages 186–203, 2025.
- [72]Kai Wang, Siqiang Luo, Caihua Shan, and Yifei Shen. Towards graph foundation models: Training on knowledge graphs enables transferability to general graphs. 2024. URL https://api.semanticscholar.org/CorpusID:278636253.arxiv.org/abs/2410.12609
- [73]Zehong Wang, Zheyuan Zhang, Nitesh V Chawla, Chuxu Zhang, and Yanfang Ye. Gft: Graph foundation model with transferable tree vocabulary. Advances in neural information processing systems, 37:107403–107443, 2024.arxiv.org/abs/2411.06070
- [74]Zehong Wang, Zheyuan Liu, Tianyi Ma, Jiazheng Li, Zheyuan Zhang, Xingbo Fu, Yiyang Li, Zhengqing Yuan, Wei Song, Yijun Ma, Qingkai Zeng, Xiusi Chen, Jianan Zhao, Jun-dong Li, Meng Jiang, Pietro Liò, Nitesh V. Chawla, Chuxu Zhang, and Yanfang Ye. Graph foundation models: A comprehensive survey. ArXiv, abs/2505.15116, 2025. URL https://api.semanticscholar.org/CorpusID:278783050.arxiv.org/abs/2505.15116
- [75]Ralph S. Woodruff. Confidence intervals for medians and other position measures. Journal of the American Statistical Association, 47(260):635–646, 1952.DOI
- [76]Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations (ICLR), 2019.arxiv.org/abs/1810.00826
- [77]Ji Xue, Feng Yan, Robert Birke, Lydia Y Chen, Thomas Scherer, and Evgenia Smirni. Practise: Robust prediction of data center time series. In 2015 11th International Conference on Network and Service Management (CNSM), pages 126–134. IEEE, 2015.DOI
- [78]Yitao Yang, Yangtao Deng, Yifan Xiong, Baochun Li, Hong Xu, and Peng Cheng. AidaI: Automated incident diagnosis for ai workloads in the cloud. arXiv preprint arXiv:2506.01481, 2025.arxiv.org/abs/2506.01481
- [79]Ze Yang, Yihong Jin, Juntian Liu, and Xinhe Xu. An intelligent fault self-healing mechanism for cloud ai systems via integration of large language models and deep reinforcement learning. 2025 8th International Conference on Advanced Electronic Materials, Computers and Software Engineering (AEMCSE), pages 344–347, 2025. URL https://api.semanticscholar.org/CorpusID:279251043.arxiv.org/abs/2506.07411
- [80]Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. Do transformers really perform badly for graph representation? In Neural Information Processing Systems, 2021. URL https://api.semanticscholar.org/CorpusID:265104899.
- [81]Minji Yoon, John Palowitch, Dustin Zelle, Ziniu Hu, Ruslan Salakhutdinov, and Bryan Perozzi. Zero-shot transfer learning within a heterogeneous graph via knowledge transfer networks. Advances in Neural Information Processing Systems, 35:27347–27359, 2022.arxiv.org/abs/2203.02018
- [82]Guangba Yu, Gou Tan, Haojia Huang, Zhenyu Zhang, Pengfei Chen, Roberto Natella, Zibin Zheng, and Michael R Lyu. A survey on failure analysis and fault injection in AI systems. ACM Transactions on Software Engineering and Methodology, 35(1):1–42, 2026.arxiv.org/abs/2407.00125
- [83]Yusheng Zheng, Wenan Mao, Shuyi Cheng, Fuqiu Feng, Guangshui Li, Zhaoyan Liao, Yongzhuo Huang, Zhenwei Xiao, Yuqing Li, Andi Quinn, et al. SysOM-AI: Continuous cross-layer performance diagnosis for production ai training. arXiv preprint arXiv:2603.29235, 2026.arxiv.org/abs/2603.29235




