WorksPhysics and Astronomy
Human Perception-Inspired Grain Segmentation Refinement Using Conditional Random Fields
Abstract
Automated detection of grain boundaries (GBs) in electron microscope images of polycrystalline materials could help accelerate the nanoscale characterization of myriad engineering materials and novel materials under scientific research. Accurate segmentation of interconnected line networks, such as GBs in polycrystalline material microstructures, poses a significant challenge due to the fragmented masks produced by conventional computer vision (CV) algorithms, including convolutional neural networks. These algorithms struggle with thin masks, often necessitating post-processing for effective contour closure and continuity. Previous approaches in this domain have typically relied on custom post-processing techniques that are problem-specific and heavily dependent on the quality of the mask obtained from a CV algorithm. Addressing this issue, this paper introduces a fast, high-fidelity post-processing technique that is universally applicable to segmentation masks of interconnected line networks. Leveraging domain knowledge about grain boundary connectivity, this method employs conditional random fields and perceptual grouping rules to refine segmentation masks of any image with a discernible grain structure. This approach significantly enhances segmentation mask accuracy by correctly reconstructing fragmented GBs in electron microscopy images of a polycrystalline oxide. The refinement improves the statistical representation of the microstructure, reflected by a 51 % improvement in a grain alignment metric that provides a more physically meaningful assessment of complex microstructures than conventional metrics. This method enables rapid and accurate characterization, facilitating an unprecedented level of data analysis and improving the understanding of GB networks, making it suitable for a range of disciplines where precise segmentation of interconnected line networks is essential.
Introduction
Integrating computer vision with electron microscopy has significantly enhanced efficiency in materials science by speeding up traditionally laborious and time-consuming processes [71]. The advancement of (scanning) transmission electron microscopy (S)TEM is pivotal in examining grain boundary (GB) networks and other microstructural features in hard materials like metals and ceramics [72]. These techniques yield insights into structural characteristics such as (dis)order [75], dislocations [56], phase transformations and GB segregation [53]. GB networks, often statistically represented [74], require extensive data for analysis. Revealing large-scale information like average grain sizes facilitates high-throughput experiments [80] aimed at elucidating local GB properties [72] and atomic (dis)order [75]. However, developing these techniques is challenging due to the complex nature of grain structures and their boundaries in ceramics, metals, and composites [68, 69], involving issues like managing overlapping grains [49] and deciphering defects of different dimensionality such as GBs, triple junctions, and nodes [6]. This complexity is accentuated when analyzing GB networks, influenced by factors like crystallographic orientations, interfacial segregation, and varied local atomic environments, affecting the mechanical, electrical, chemical, and magnetic behavior of materials [1, 2, 3, 5, 10, 11, 12, 13, 37, 51, 63, 78]. GBs range from ordered high-symmetry structures to high-energy disordered configurations [24], with networks exhibiting intricate relationships due to energetic competition between interfacial planes, atomic sites, and solute-solute interactions [4, 26, 70]. Moreover, local ordering and local hardening can be influenced by solute segregation to GBs [6, 45]. Therefore, developing predictive techniques is challenging, considering the multifaceted role of GBs where local atomic environments, segregation behaviors, and solute interactions are closely connected. This complexity often results in (S)TEM image contrasts, posing a unique challenge for conventional segmentation methods in identifying GBs.
Addressing this, semantic segmentation – a computer vision strategy categorizing each pixel into a class [46] – emerges as a promising approach for autonomous grain segmentation. Historically, two techniques have been prominent: traditional image processing and modern computer vision techniques. Traditional methods focus on low-level image details, using techniques like thresholding [54], morphological processing [21], edge detection with preset filters [29], unsupervised machine learning clustering [73], watershed transformation [62], and region-growing [61]. Their effectiveness often depends on image quality, influenced by factors like resolution, color balance, brightness, and gradient similarity [44]. Conversely, modern computer vision approaches, particularly convolutional neural networks (CNNs), provide a more sophisticated, high-level grain representation [23, 27, 55, 58]. Architectures like DeepLab [18], Mask R-CNN [30], and U-Net [60] are employed for their adaptability to different image resolutions and consistent segmentation capabilities.
However, CNNs’ large perception fields limit their ability to produce accurate pixel-level labels [46], which poses a problem in an interconnected grain network where the segmentation mask is a few pixels wide with a label assigned to each pixel (e.g., 1: GB, 0: grain interior). Metrics like intersection-over-union (IoU) and Dice similarity coefficient (DSC) are used to assess the alignment of predicted masks with ground truth. But, in thin masks, even minor 1-pixel deviations significantly affect accuracy [19]. Researchers have explored various categories of post-processing methods to enhance the segmentation accuracy of thin, interconnected line structures like GBs. One prominent category involves probabilistic models, which refine initial segmentation masks by enforcing spatial consistency. For instance, one such approach has been successfully applied to road network extraction from aerial imagery by smoothing fragmented predictions and maintaining continuity [9]. This was achieved through an iterative search process, where a CNN-driven decision function directly generated the road network graph from the CNN’s output. In another application, Dulau et al. [25] demonstrated this by improving the continuity of retinal blood vessel maps, by removing misclassified pixels erroneously identified as retinal vessels, and by rejoining vessel segments that are correctly identified yet remain separated. However, these techniques typically rely on heuristic rules and can struggle with noisy or incomplete initial predictions.
Morphological and perceptual grouping approaches represent another prevalent set of post-processing methods. Morphological operations, such as dilation and skeletonization, have been widely used for tasks like cleaning up segmentation masks or connecting small gaps in biomedical vessel segmentation [40]. Perceptual grouping methods, like tensor voting, have been employed effectively to reconnect fragmented segments in pavement crack detection [81]. Despite their utility, these methods often require extensive manual tuning of parameters, limiting their general applicability across diverse datasets. In contrast, model-specific integrated post-processing methods, like Li et al. [42], have demonstrated remarkable accuracy by incorporating generative adversarial networks and multi-task learning explicitly designed for GB segmentation. Furthermore, Patrick et al. [52] introduced an automated grain boundary detection framework based on the U-Net architecture for BF TEM images that achieves high segmentation accuracy with minimal manual intervention, improving processing throughput. Yet, the specialized nature and reliance on tailored parameters significantly restrict their general use beyond specific materials or image types. Main challenges include connecting adjacent pixels and removing isolated ones, relying on user-defined thresholds. These methods often miss crucial constraints related to GBs, essential for accurate segmentation of complex grain structures.
Here, this work aims to address the gap in semantic segmentation for (S)TEM images of polycrystalline materials by introducing a versatile post-processing method to refine interconnected GB network masks. This method, based on insights into GB connectivity [43] and the interconnected nature of GB networks [14], is designed for broad applicability across different crystalline materials. At its core, it employs segmentation masks from computer vision models to generate conditional probability maps using conditional random fields (CRFs), bridging gaps left by CNNs. The required feature functions for CRFs are selected based on perceptual grouping rules [50]. Applicable to any crystalline material with discernible grain structure, and adaptable to other domains with interconnected line networks, this method can be used as a post-processing step for various grain segmentation masks from imaging systems and vision algorithms, enabling real-time segmentation refinement. It marks a significant advancement for materials scientists, allowing rapid and precise GB segmentation in complex microstructures with poorly resolved boundaries, and enabling previously inaccessible large-scale data analysis, greatly enhancing statistical representation of GB networks.
Methods
The systematic approach for grain segmentation refinement, aimed at addressing the limitations of conventional computer vision algorithms, is outlined in the subsequent sections. Initially, a computer vision model is developed, serving as a preliminary step to generate segmentation masks. These thin segmentation masks which include fragmented sections form the foundation for applying the proposed hierarchical CRF method. This method, along with perceptual grouping principles, is introduced to address various aspects of grain boundary properties and connectivity. For a given image, the solution that maximizes the conditional probability obtained from the hierarchical CRF corresponds to a label configuration that determines points of connection. Subsequently, a novel path tracing algorithm is introduced, designed to connect fragmented segments using the label configuration derived from the CRF predictions.
Computer vision model development
The input dataset for model training consisted of high-angle annular dark field and annular bright field STEM images recorded with a Nion UltraSTEM100 aberration-corrected STEM operating at 60 kV. The material is an electrically conducting polycrystalline oxide, fluorite , synthesized by pulsed laser deposition as a layer of nearly uniform thickness ( nm) atop an amorphous silicon nitride free standing substrate.
The images in the dataset are digitized and manually annotated by tracing GBs by hand using a digital drawing tablet, a process widely utilized in materials science, to assess grain sizes and network structures for subsequent analysis of material properties. GBs marked on the images constitute the computer vision ground truth (CVGT), serving as targets for model training. To minimize human operator bias, both annular dark field and bright field images of the same areas were annotated at different times. Regions marked as interfaces in both image types reinforce the trained model’s predictions of interface pixels.
A pre-processing pipeline is implemented to enhance data diversity and simulate realistic experimental conditions. The dataset is initially partitioned at the level of the original images into training and validation sets using a 70–30 split. Each image is subsequently subdivided into non-overlapping patches of pixels. Each patch typically contains approximately 64 grains, capturing the inherent repetitive microstructural features of the material.
To further improve model generalization and simulate experimental variability, an augmentation pipeline is applied exclusively to the training patches. This pipeline performs random brightness adjustments (factors ranging from 0.8 to 1.2), Gaussian blurring (with sigma values between 0.5 and 1.5), and rotations (between and ). By applying 20 random augmentations per training patch, the effective number of training samples is expanded to 4480, while the validation set remained unaltered at 144 patches to ensure an unbiased evaluation of model performance.
A computer vision algorithm for image segmentation is developed using a modified U-Net architecture [60], with VGG16 model [66] serving as the encoder backbone. This architecture, effective for varying image resolutions, features a symmetric encoder-decoder structure [60] with skip connections to preserve key details lost during downscaling [31]. It consists of a contracting path capturing low-level features while reducing spatial size and increasing channels, and an expanding path that upscales these features to original size, reducing channels and capturing high-level features for accurate segmentation [60]. The model includes four decoder blocks, each with a 2D convolution layer matching the corresponding encoder layer’s channels, followed by concatenation with the encoder’s skip connection and two convolutional blocks with batch normalization and ReLU activation. The final output uses a sigmoid function. Compiled with Adam optimizer [36] with a learning rate of and a custom loss function (), the model employs early stopping based on validation loss improvement, monitoring metrics like DSC, IoU and binary accuracy.
The objective of this work is not to achieve the highest possible segmentation accuracy but rather to demonstrate that the post-processing model is learning meaningful features rather than overfitting. To this end, no extensive hyperparameter tuning or further optimization techniques, such as learning rate scheduling, ensemble methods, or advanced regularization, were pursued. Instead, a 3-fold cross-validation strategy is employed on the augmented training set to robustly assess the model’s performance. This approach ensures that the model achieves a good enough performance level, with stable validation results indicating that it is effectively capturing the key features of the GBs without overfitting.
A hierarchical CRF architecture
When labeling sequence is critical, such as for interface pixels, CRFs emerge as a versatile tool. As a discriminative model, CRFs calculate the conditional probability of a pixel or segment having the same label given a sequence of observations [67]. CRFs require feature functions, mathematical representations of feature relationships [67]. An essential characteristic of CRFs, rooted in their Gibbs distribution formulation, is the concept of conditional independence. This implies that two pixels or segments without a direct interaction (as defined by the CRF graph structure) become conditionally independent once the state of their neighborhoods is fixed. Thus, distant pixels, though possibly globally correlated through intermediate connections, are directly independent in the conditional sense. Practically, this means the CRF explicitly models local and selected long-range interactions, making computational complexity manageable while preserving meaningful global coherence.
To incorporate domain knowledge accompanying the CRF method, the feature functions can be selected based on perceptual grouping principles. These principles, derived from Gestalt laws of perceptual organization, describe how humans intuitively perceive and interpret the visual world [50] and are relevant to grain segmentation refinement, as illustrated in Figure 1. Key principles include:

Figure 1. Overview of the utilization of the Gestalt principles of perceptual grouping. (a) Proximity, (b) similarity, (c) closure, (d) good figure, and (e) continuity principles are satisfied through the use of (f) pixel-based CRF, and (g) segment-based CRFs resulting in (h) a refined segmentation. Each subplot provides a distinct visualization or method that contributes to the overarching theme of the paper, the robust analysis of GB networks.
The Law of Proximity, indicating that elements close together are perceived as a group. In Figure 1a, clustered dots exemplify this, aiding in identifying nearby pixels or segments as part of the same GB, especially areas of closely packed or intersecting boundaries.
The Law of Similarity, where similar elements are seen as related. Figure 1b shows this with dots of the same color and shape, grouping pixels or segments with similar characteristics for continuous GB identification.
The Law of Closure, involving the mind’s tendency to complete incomplete figures, as in Figure 1c. This helps in filling gaps in GB
networks that appear broken, addressing challenges posed by poorly resolved and complex structures in TEM images.
The Law of Good Figure (or Prägnanz), where complex images are perceived in their simplest form, as seen in Figure 1d. This aligns with GBs’ tendency to minimize energy by adopting simpler shapes, reflecting the natural formation and evolution of GBs in materials.
The Law of Continuity, favoring smooth patterns. Figure 1e demonstrates elements aligned in a line or curve as a cohesive group, essential for maintaining continuous and smooth paths consistent with the physical properties of grain structures.
Fully-connected independent pixelwise classification, often leading to patchy segmentation masks due to disregarding pixel relationships, is computationally demanding [8]. CRFs are preferred in such contexts as they effectively incorporate spatial relationships between pixels using pairwise costs. Early models primarily utilized grid graphs for defining these costs, which restricted information transfer among adjacent pixels [65]. However, the introduction of DenseCRF architectures allowed for fully connected graphs and more expressive pairwise potentials [22, 38]. Higher-order CRFs for contour completion bring three key benefits to grain segmentation: (i) they foster label continuity among spatially and intensity-wise close pixels, (ii) facilitate modeling of long-range interactions, and (iii) use linear inequalities for modeling extension and continuity constraints.
In this work, a hierarchical methodology for object recognition and segmentation is introduced, combining pixel-based and segment-based definitions synergistically. The novelty of this approach lies in its hypergraph architecture, which integrates a DenseCRF for low-level pixel features with a higher-order, segment-based CRF that enforces domain-specific knowledge through perceptual grouping rules. This synergistic combination allows the model to refine fragmented boundaries by considering both local pixel evidence and global structural coherence, a capability not explicitly present in standard DenseCRF or higher order CRF models alone. The architecture is designed as a hypergraph, with vertices as individual pixels and hyperedges as segments. Inter-segment connections are construed as edges between hyperedges. Initially, a DenseCRF architecture [38] processes low-level pixel features (Figure 1(a), (Figure 1b), and (Figure 1f)), which then inform the higher-level segment-based features, as illustrated in Figure 1(c)-(Figure 1e), and (Figure 1g). This layered strategy allows for the integration of both local and global information, adhering to perceptual grouping principles.
The conditional probability of segments, which describes the probability distribution over all possible label configurations, is determined by minimizing Gibbs energy, comprising unary, pairwise, and higher-order terms. The unary potential encodes the likelihood of individual pixels belonging to specific segments, either complete or broken. This potential is derived from the pre-trained CNNs that provide scores for each pixel, essentially constituting an initial coarse-grained classification. Local consistency is subsequently enforced through edge-preserving bilateral and Gaussian blur filters, yielding perceptually coherent regions by evaluating neighboring pixels’ features like color and proximity, as shown in Figure 1(a) and (Figure 1b). This effectively distinguishes segments in densely packed GBs, with non-conforming pixels eliminated, depicted in Figure 1(a), (Figure 1b), and (Figure 1f) with red circles. Subsequently, a graph-based technique is employed to identify completion segments, which represent potential connections between broken segments. Due to computational limitations, only a subset of these completion segments, selected based on predetermined criteria, are deemed viable segments for further analysis. The various segment types are visually differentiated in Figure 1c, with complete, broken, and completion segments shown in white, red, and yellow, respectively.
The pairwise potential, designed to capture long-range interactions, functions effectively on identified segments to maintain global consistency. It enforces completion and extension constraints, as described in a previous contour completion model [79], aligning with the principle of closure illustrated in Figure 1c. The completion constraint activates a completion segment only when its adjacent broken segment is active. Simultaneously, the extension constraint ensures that an active broken segment has at least one neighboring active completion segment, preserving both the continuity and integrity of the GB structure.
A higher-order potential incorporates specific visual characteristics like contour sharpness, smoother segment transitions, angular differences between segments, and segment lengths. The model’s complexity is managed by considering additional features such as the total effective length of connected segments and the angular deviation from ideal connection angles. This optimizes simpler, more stable GB forms, adhering to the law of good figure, reflecting the natural evolution of GBs to minimize energy and simplify complex structures for perceptual coherence, as shown in Figure 1d.
In this composite architecture, the hypergraph method efficiently captures intricate relationships between pixels and segments, enabling a thorough representation of spatial and geometric features in the image. The primary goal is to achieve a binary label configuration that maximizes the conditional probability, representing the CRF’s maximum a posteriori probability estimate. This poses a combinatorial optimization challenge, where the ideal solution is the label configuration maximizing this probability. This optimization problem can be posed as a mixed-integer linear programming (MILP) problem, which can then be solved to obtain the optimal label configuration.
Identifying the optimal label configuration involves determining local weights for interface features and global weights for unary, pairwise, and higher-order potentials. The optimal configuration is that which most closely approximates the ground truth, leading to an NP-hard combinatorial optimization problem. Being NP-hard, it is defined by the property that while the validity of a solution, like a label configuration, can be efficiently verified via the MILP equation, there is no polynomial-time algorithm for optimal resolution. Consequently, exact solutions to such NP-hard problems are computationally infeasible [77], especially in images with hundreds of broken and completion segments, typical of fragmented GB networks.
To address this challenge, this study employs differential evolution to minimize the objective function, achieving a near-optimal solution. As an evolutionary algorithm and global optimization method, differential evolution iteratively refines candidate solutions, adjusting their weights based on performance metrics within set bounds [57]. This process derives weights for the optimal label configuration by repeatedly solving the MILP with varied weight combinations. The resulting optimal label configuration, indicated by activated segments (marked as 1 by the CRF), facilitates the smooth connection of completion segments. This approach aligns with the law of continuity, essential for ensuring continuous, logically flowing GBs in the segmentation process, consistent with the physical nature of grain structures, as depicted in Figure 1e.
Applying the hierarchical CRF to probabilistic image labeling
The conditional probability, , describes the probability distribution over all possible label configurations given the observed data . refers to the entire set of observed data over the complete image, containing all of the features (e.g., pixel intensities, color channels, spatial segment information and any other characteristics of the images) that are used to determine the labels . refers to the set of all labels or the complete labeling configuration for an entire image, that is represented as a vector containing the labels for each pixel or segment. On the other hand, refers to the label of an individual unit, such as a single pixel or segment. Essentially, each is an element of the set . and refer to the label configurations assigned to pixels and segments that are part of a set, respectively. The indices , , and represent the different entities in the image, where is the pixel or segment that is currently being labeled, and and represent other pixels or segments in the same clique as . The relationships between , , and are encapsulated in the pairwise and higher-order terms of the CRF, capturing local and higher-order interactions in the image. The overall probability distribution is defined as:
and
where , , and are the unary, pairwise, and higher-order Gibbs energies, respectively. These energies are calculated based on the labeling of segments , , and , not just individual pixels. is a partition function that ensures a normalized probability distribution (i.e., between 0 and 1), given by:
The unary energy term in Eq. (2), , is derived from the pixel-based CRF where the unary term is generated by a CNN and the pairwise term is obtained from a Gaussian kernel. Explicitly, for the unary term, the segment-based can be defined as:
Here, represent a global weight and represents a classifier, which identifies pixels as belonging to one of two segment types: complete or broken segments. The set of segments is represented by , which is the union of two segment sets; complete segments, , and broken segments, , respectively. These sets are utilized to create segment cliques, which are groups of interconnected segments categorized into pair cliques, and completion and broken cliques. Sample cliques from each type are shown in Figure 2a-Figure 2c, where broken segments are represented in red and completion segments in yellow. The black pixels represent background pixels, gray pixels represent foreground pixels, and white pixels correspond to isolated points. These cliques represent different configurations of segments based on their relationship and interaction with each other. A pair clique, , shown in Figure 2a, is comprised of a pair of broken and completion segments. A completion clique, , shown in Figure 2b, describes a completion segment and its neighboring broken segments. Finally, a broken clique, , shown in Figure 2c, contains all completion segments connecting to either end of a broken segment.

Figure 2. Visualization of cliques and classification processes. (a) Pair Clique (): Interaction between a broken (red) and completion (yellow) segment. (b) Completion Clique (): A completion segment with its neighboring broken segments. (c) Broken Clique (): Completion segments connecting to a central broken segment. (d) Post-classification detailing broken segments, triple junctions, and natural curving angles in a segmentation mask. (e) All possible completion segments generated between isolated points. (f) Selection of viable completion segments from the generated possibilities. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
At the classification stage, in addition to the broken and complete segments, triple junctions, isolated points, and the natural curving angles of segments are identified, which are shown in Figure 2d for a sample section of the segmentation mask. In order to obtain completion segments, first all possible completion segments between all pairs of isolated points are generated, which is shown in Figure 2e. Then, viable completion segments, which are defined as non-occluding completion segments below a predetermined length, are down selected from all possible segments, which are shown in Figure 2f. Hence the classifier maps the unary and pairwise energies of pixels constituting a segment, , to the segment’s unary energy , where is given by:
where the pairwise-pixel features are given by the following equations:
and
Here, is a label compatibility function that follows the Potts model for cost assignments, which states that the cost is non-zero only if the pixels are labeled differently. In other words, if , and 0 otherwise. For a binary segmentation task, this model punishes similar pixels with different labels. Pairwise terms are calculated using two features involving Gaussian and bilateral filters, which are instrumental in satisfying the similarity and proximity principles. These principles suggest that pixels that are closely related in intensity and proximity tend to be grouped together. , , and are the parameters for the Gaussian and bilateral filters that can be adjusted, and and are the corresponding local weights.
The significance of closure principle in visual perception, while challenging to encapsulate through local potential functions, is approximated in the study by Ming et al. [79] via a series of localized connectedness constraints. Within a segment’s terminal point, two kinds of connectedness constraints are defined, each defined with a series of linear inequalities:
The Completion Constraint is characterized by the limitation that a completion segment can only be activated if its adjacent broken segment is also active, which requires the following inequality to be satisfied: .
The Extension Constraint is defined by the condition that when a broken segment is active, at least one neighboring completion segment must also be active for possible extension. This is expressed as the inequality: .
These constraints can be represented in the pairwise term for segments with the following expression:
where is a large constant, which is large enough to ensure that Eq. 1 results in a zero-probability configuration (i.e., ) in the case that these inequalities are not satisfied.
In line with the law of good figure, which generally favors simpler, more coherent shapes, the higher-order term is defined to obtain the best possible label configuration that is grounded in the domain expertise. Similar to the models in Ref. [22, 79], different potentials are utilized to define a higher-order Gibbs energy. In this work, the higher-order term is represented as the sum of an interface potential and a complexity potential:
where and are global weights. The interface potential is defined as:
where are the local weights for , which represent the interface features. There are seven interface features, the first four of which are obtained from Ref. [79], and an additional three are considered in this work. The initial four features are: (1) effective corner distance, optimized for capturing sharp turns in contours; (2) effective smooth distance, designed for capturing smoother transitions between segments; (3) angular completion, emphasizes geometric shapes through the sum of angles; and (4) angular difference, useful for identifying parallel or nearly parallel segments. The final three features pertain to the lengths of the two broken segments the one completion segment that forms the clique. Finally, the complexity potential is given by:
where are the local weights for , which includes two features representing the global complexity. These two features are: (i) , which is the total effective length of the connected segments, and (ii) , which represents the angle deviation from the ideal curving angle and the connection angle, effectively controlling the model’s overall complexity.
The constraints introduced in Eq. 7 posit that and can both be active at the same time if and only if is also active, therefore simplifying the higher-order term to:
which in turn allows framing the combinatorial optimization problem as a MILP problem, for which the final energy function to be minimized takes the form:
subject to and . Details pertaining to the implementation of pixel-based and segment-based CRFs are provided in Supplementary Notes 4 and 5, respectively.
Tracing smooth paths
Building on the probabilistic image labeling achieved through the
Hierarchical CRF method, the next step in ensuring seamless connectivity involves tracing smooth, efficient paths. Existing methods like the stochastic completion fields method [59, 76] can trace smooth paths between points, yet they lack guaranteed solutions and can be computationally intensive. Hence, in this work, where the identification of GBs for subsequent chemical analysis is time-sensitive, a cost-based pathfinding algorithm is utilized to find the lowest-cost shortest path between the endpoints of a completion segment, denoted as the source and the sink. Cost maps are generated using isolated points and curving angles of each segment in the ideal label configuration, as shown in Figure 3a. Costs are derived from probability distributions based on the natural curving angles of adjacent broken segments (Figure 3b), aiding in connectivity assumptions at interfaces. This method mirrors GBs’ energy-minimization tendency, favoring shorter, less costly paths similar to how GBs naturally form by reducing their area (length in a projection) to lower their energy state. The selected pathfinding algorithm is the algorithm, a graph traversal and path search algorithm [32] that employs a heuristic function, here chosen to be the Chebyshev distance with 8-way directionality, with the rationale for this selection being elaborated in Supplementary Note 1. This algorithm starts from the source, moving towards the sink while tracking path and associated cost. Paths are reconstructed by backtracking the lowest-cost shortest paths identified by , as in Figure 3c. These paths, shown in Figure 3g, connect broken segments in fragmented masks, resulting in refined segmentation depicted in Figure 3h, reflecting GBs’ natural energy-minimizing behavior. Further details on the pathfinding algorithm can be found in Supplementary Note 1.

Figure 3. A cost-based pathfinding algorithm. (a) The ideal label configuration with identified broken and completion segments, alongside curving angles. (b) The Gaussian cost map with varying color intensities and specific cost values. The inset illustrates the different costs assigned to pixels drawn from a 2D exact Gaussian distribution, based on the curving angle. (c) The lowest-cost shortest path is determined post-application of the pathfinding algorithm.
Results
To demonstrate this methodology, fragmented segmentation masks are generated using the trained computer vision algorithm. The annotated images serve as ground truth for model training, labeled as CVGT to differentiate from fragmented segmentation ground truth (FSGT). CVGT assesses the computer vision model’s performance, while FSGT evaluates segmentation refinement. To enhance the robustness and generalizability of the model, data augmentation methods were carefully selected. Brightness adjustment is used to simulate variations in illumination, Gaussian blur to mimic slight focus variations common in microscopy, and rotation to ensure the model’s insensitivity to orientation changes. These augmentations aim to produce a more versatile and adaptable computer vision algorithm. A sample of these augmentations is displayed in Figure 4a. The trained model, described in the Methods section, is then utilized to generate segmentation masks from the input images. The model effectively delineates boundaries in both validation and unseen images, as shown in Figure 4(b) and (Figure 4c), and in comparisons between CVGTs and predicted masks. It successfully generates segmentation masks for images not included in the training, performing better on images with fewer atomic features and pronounced contrast.

Figure 4. Computer vision model training and performance. (a) One of the original input images and its corresponding data augmentations. (b) Images from the validation set, highlighting the computer vision ground truth (CVGT) and model-predicted fragmented segmentation masks. (c) The model’s predictions on unseen images. (d) Plots of the intersection-over-union (IoU) and dice similarity coefficient (DSC) metrics along with the loss function (1-DSC) across training epochs for each fold.
In this study, a 3-fold cross-validation strategy was employed during network training over 50 epochs. The aggregated metrics across folds indicate an average training IoU of approximately 0.83 and a corresponding validation IoU near 0.67, while the training DSC approached 0.91 and the validation DSC stabilized at around 0.79, as shown in Figure 4d. These similar performance values across folds reflect that the model is reliably capturing the key features of the grain boundaries and generalizing well to unseen data. Specifically, the consistency of these metrics across different splits suggests that the model is not overly sensitive to the specific partitioning of the data, which is a strong indication of robust learning rather than overfitting.
In addition, an alternative training approach was explored that consolidated all training data and extended the training duration. Although this method yielded higher training scores—a result of the more intensive training—the validation scores remained comparable to those of the cross-validation model. While both approaches deliver similar validation performance, the cross-validation model confirms the robustness of the segmentation pipeline, and the alternative approach is particularly valuable for demonstration purposes because it produces sharper, more detailed segmentation outputs. This enhanced detail facilitates a clearer presentation of the efficacy of the subsequent post-processing stage in rectifying fragmented masks.
The performance of the model aligns with benchmarks in thin segmentation mask tasks, comparable to a DSC of 0.7 [34] reported by Jahangard et al. for retinal blood vessel segmentation. Given the focus of this work on refining the fragmented segmentation mask, further optimizations like hyperparameter tuning are not pursued. The model’s predictions on full-resolution images, critical for analyzing detailed material structures potentially lost or distorted in lower resolutions, demonstrate its efficiency in processing high-resolution images.
On the fragmented segmentation masks, pixel-based CRFs are applied, followed by the creation of the FSGT. FSGT is formed by manually labeling each identified completion and broken segment, following specific principles including consistent judgement across all images, marking the connection segments only if the natural curvatures of the broken segments appear to intersect and ensure non-occluding connections. These FSGTs aid in learning optimal weights and evaluating the proposed method’s performance. In a sample image, Figure 5a displays all segments identified by the algorithm, Figure 5b the manually labeled FSGT, and Figure 5c the best solution from the differential evolution algorithm, which corresponds to the minimum difference in labeling configurations between the FSGT and MILP solution. This accuracy is calculated as the percentage of correctly classified segments (i.e., true positives and true negatives) from the total number of viable segments. Insets in Figure 5 magnify portions where the CRF accurately predicts most segments, though as noted, some discontinuities may persist where the probabilistic evidence for a connection is insufficient. Additional magnified views are shown in Fig. S2. Figure 5c illustrates newly identified grains from connected segments, enhancing segmentation accuracy, as the subsequent watershed algorithm identifies grains by continuous lines forming the boundary of closed geometric shapes.

Figure 5. Detailed examination of the segment-based CRF labeling. (a) All viable segments identified by the algorithm with broken (red) and completion (yellow) segments. (b) The manually labeled fragmented segmentation ground truth (FSGT). (c) The CRF predicted label configuration, illustrating the optimal segment connections post-differential evolution algorithm. Insets in each subplot offer a magnified view of selected areas, elucidating the finer details of segmentation. The image scale is consistent with that shown in Figure 4b. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
Watershed segmentation
A marker-based watershed segmentation technique, based on geographical watershed principles [16, 62], is employed for image segmentation. This method converts the image into a topographical map with pixel values representing altitudes, applying the watershed transform to create catchment basins associated with distinct markers. The basin boundaries, as shown in Figure 6a, segment the image into distinct regions or grains. However, broken lines in the mask, such as those visible in Figure 6a, can result in suboptimal segmentation. The grains in these plots appear as randomly colored shapes, except for the region depicted in the insets, where grain colors are matched manually for better visualization. More details about this method can be found in Supplementary Note 2. This segmentation is applied to three image types: (1) the original segmentation mask before post-processing (Figure 6a), (2) the FSGT (Figure 6b), and (3) the CRF solution (Figure 6c). The figure shows that broken segments without viable completion segments are removed, and sections are smoothly connected, thereby enhancing segmentation. Insets in Figure 6 demonstrate the connections shown in Figure 5c results in better grain segmentation, impacting the statistical representation of GB networks through improvements in grain shape and area. Additional magnified views are shown in Fig. S3. These segmented images are then used to evaluate the performance of the post-processing procedure.

Figure 6. Image segmentation using marker-based watershed technique. (a) Original segmentation mask before post-processing with visible broken segments. (b) Fragmented segmentation ground truth (FSGT) image that highlights the connections between the disconnected segments. (c) CRF solution corresponding to the optimal label configuration after post-processing, showing removal of dangling broken segments and smoother connections between sections. The grains are randomly colored, except for the insets. Insets in each subplot provide a magnified view, emphasizing the differences in grain identification and segmentation performance. The image scale is consistent with that shown in Figure 4b.
Performance evaluation
In the segmentation performance evaluation, the post-processing procedure consistently improved various metrics, as shown in Table 1. Pixel accuracy, measuring correctly classified pixels against the total number, increased by 0.18 % in the validation set from an already high baseline of 0.997 to 0.999, indicating near-perfect classification accuracy. To better assess segmentation masks, metrics such as IoU and DSC are often employed [19]. IoU and DSC metrics exhibited an increase of 9.15 % and 5.19 % in the validation set due to post-processing, respectively. These improvements bring the final IoU and DSC values to 92.5 % and 96.1 % representing significant enhancements in segmentation accuracy, particularly for tasks involving complex grain structures. Calculation details for these metrics are provided in Supplementary Note 3. Additionally, grain-specific metrics, like the number of grains, , and average grain size, , also demonstrated notable improvements, detailed in Table 2. These enhancements were calculated as percentage improvements over the FSGT baseline, based on their relative differences.
| Metric | Training set | Validation set | ||||
| FSGT / before* | FSGT / after* | Percent change | FSGT / before* | FSGT / after* | Percent change | |
| Accuracy | 0.998 | 0.999 | 0.18 % | 0.997 | 0.999 | 0.18 % |
| IoU | 0.911 | 0.979 | 6.78 % | 0.833 | 0.925 | 9.15 % |
| DSC | 0.954 | 0.989 | 3.58 % | 0.909 | 0.961 | 5.19 % |
| Precision | 0.929 | 0.996 | 6.66 % | 0.853 | 0.967 | 11.36 % |
| Recall | 0.979 | 0.983 | 0.39 % | 0.973 | 0.955 | −1.77 % |
| Grain alignment | 777.8 | 174.2 | 77.6 % | 4062.7 | 2007.7 | 50.6 % |
Table 1. Comparison of segmentation performance evaluation metrics for both training and validation sets, highlighting the improvements achieved through post-processing.
| Metric | Training set | Validation set | ||||||
| FSGT* | Before* | After* | Percent change | FSGT* | Before* | After* | Percent change | |
| 904 | 797 | 893 | 10.62 % | 1518 | 1206 | 1520 | 20.69 % | |
| 4309.0 | 4812.8 | 4429.2 | 8.90 % | 10,864.0 | 13,293.4 | 11,180.7 | 19.45 % |
Table 2. Metrics specific to grain attributes for both training and validation sets, illustrating the number of grains, and average size of identified grains, and their respective percent changes after post-processing.
An analysis of confusion matrices is presented in Figure 7a for the training set and Figure 7b for the validation set. In the training dataset, the CRF correctly predicts 462 out of 577 segments, achieving approximately 80 % accuracy. For the validation set, it accurately predicts 3090 out of 3937 segments, a 79 % success rate. Regarding time efficiency, manual labeling to create the FSGT is a necessary but time-intensive prerequisite, taking roughly 41 min per 1000 segments. In contrast, the automated CRF method labeled 1000 segments in about 15 milliseconds on an Intel Core i7-12700K 12-Core Processor. This five-order-of-magnitude speed-up is not intended to diminish the crucial role of manual annotation for training and validation, but rather to highlight the framework’s potential for high-throughput analysis where such manual intervention is infeasible. Additionally, weight learning, needed once per application, completes in around 47 min. For fully automated STEM imaging and spectroscopic data acquisition in GB network chemical analysis, the CRF’s rapid processing rate would enable time-efficient implementation needed to mitigate the effects of microscope instability such as specimen drift and lens defocus.

Figure 7. Detailed evaluation metrics for segmentation performance. Confusion matrices illustrating the alignment of predicted segment labels with true labels for (a) the training set, and (b) the validation set. ‘active’ and ‘inactive’ denote segments identified as kept or discarded, respectively, highlighting the model’s classification efficacy.
A novel metric for grain alignment is developed to overcome the limitations of traditional metrics in this specific segmentation context. This metric computes the average absolute difference between the areas of closely aligned regions in two labeled images, identified based on the proximity of their centroids. A smaller value for grain alignment indicates a higher degree of similarity in terms of grain size between the two images, making it an effective measure for comparing segmented regions, especially in the context of thin segmentation masks. Significant improvements in mean grain alignment, up to 78% in training and 51% in validation sets, are observed post-segmentation. These enhancements are critical as they significantly enhance the accuracy of statistical representation of grain size and location, leading to more precise analysis of GB networks, especially in densely packed or intersecting boundary regions.
Discussion
Beyond conventional metrics
Pixel accuracy focuses on the count of foreground and background pixels and neglects their spatial distribution, which limits its utility in single-pixel wide masks. In cases where discarded broken segment pixels
The bold values indicate the percent changes in the metrics.
The bold values indicate the percent changes in the metrics.
equal the number of added completion segment pixels, pixel accuracy remains unchanged, potentially misrepresenting changes in spatial distribution. While high pixel accuracy may suggest superior performance, it can obscure spatial distribution nuances. IoU and DSC, although more nuanced, are sensitive to minor spatial variations, which is a notable limitation in thin masks. Nevertheless, improvements in these metrics
post-processing affirm their utility in assessing the alignment of segmentation masks with ground truth. Table 2 shows post-processing enhancements in actual segmented grains. Both the number of identified grains and average grain size align more closely with the ground truth post-processing, signifying not only improved alignment but also enhanced grain segmentation attributes.
These advancements indicate a more accurate representation of grain size distribution and density, crucial for assessing mechanical and physical material properties. GBs considerably affect aspects like crack propagation, corrosion resistance, and mechanical strength, highlighting the importance of these improvements.
Table 1 and the confusion matrices in Figure 7a and Figure 7b reveal a decrease in recall for the validation set, suggesting the model’s reduced sensitivity in identifying active segments post-processing. This could stem from unseen data or minor overfitting during training. These metrics primarily assess the model’s proficiency in correctly labeling pixels or segments as active or inactive. However, in grain segmentation tasks that emphasize not only accurate pixel classification but also the morphological characteristics of the grains, such metrics may not fully capture the performance.
Addressing limitations in conventional metrics, a new metric, grain alignment, is proposed for evaluating segmentation performance. This metric involves identifying centroids of corresponding regions in two labeled images. For each grain in the first image, the closest centroid grain in the second image is determined, ensuring spatial proximity-based comparison. The absolute differences in the areas of these closely aligned regions are calculated and averaged to gauge size similarity, offering a robust measure for segmented region accuracy, particularly in thin masks. While this approach provides a robust metric, it’s important to note the mean grain alignment metric difference between the training and validation dataset can be attributed to a few considerations. This discrepancy arises because accurately identifying grains requires all constituting lines to be connected. In fragmented images, unconnected lines can lead to incorrect grain identification, merging multiple grains into a single large one, and thus affecting the grain count and spatial distribution. The post-processing method rectifies this by connecting fragmented boundaries; by successfully closing a previously open boundary with a completion segment, a single, incorrectly merged grain is correctly partitioned into two or more distinct grains. This partitioning effect is the primary reason for the observed increase in the number of grains () post-processing (Table 2), bringing the count closer to the ground truth. Additionally, the post-processing’s single-pass might leave some segments incomplete. This difference is more indicative of the validation set’s complexities rather than a general limitation of the method. By identifying closely situated grains as part of the same boundary network, this method precisely represents microstructures, essential for accurate material behavior prediction, demonstrating its utility despite the noted dataset-specific challenges.
In addition to the demonstrated accuracy, the CRF’s remarkable time efficiency is particularly valuable in materials science applications requiring real-time analysis. This rapid segmentation capability, in stark contrast to the time-consuming manual labeling process, allows for immediate GB identification post-image acquisition. Crucial in countering issues like drift in TEM imaging, real-time processing ensures analyses are based on accurately segmented, current images. This not only boosts analysis reliability but also enables dynamic, in-situ experimentation and observation in materials science, where monitoring changes over time and under varying conditions is essential.
Comparing standard and specialized ground truths
The CVGT and FSGT evaluate distinct aspects of segmentation, targeting specific criteria. The CVGT focuses on essential features for initial computer vision training, whereas the FSGT addresses specialized grain segmentation requirements, including unique post-processing needs such as continuous grain lines. Consequently, comparing post-processed results directly with the CVGT is inherently flawed. The CVGT, serving as a baseline for assessing computer vision algorithm performance, varies across images and is contingent on the computer vision output’s quality. Therefore, it becomes an unstable and unreliable baseline for evaluating the post-processing procedure’s performance, which inherently refines and alters the computer vision algorithm’s initial segmentation.
Post-processing impacts metrics like IoU and DSC, influenced by the initial model’s alignment with the CVGT. Generally, good alignment enhances these metrics post-processing, whereas poor alignment diminishes them. For instance, in Table 1, using the CVGT as ground truth, IoU and DSC in the training set decrease by 0.12 % and 0.21 %, respectively, despite a 78 % improvement in grain alignment. This indicates that post-processing accentuates the initial segmentation’s adherence to the CVGT. Therefore, to isolate the specific influence of the post-processing procedure, a comparison with the FSGT is essential. Additionally, the subjectivity in annotating GBs complicates evaluations. Consistency in CVGT and FSGT annotations is difficult due to specific grain characteristics, introducing variability in ground truths and affecting metrics. This underlines the limitations of relying solely on CVGT for evaluation and the importance of including the FSGT for comprehensive assessment.
Towards advanced segmentation and cross-domain applications
Factors like hyperparameter optimization, model choice, and more training data can enhance baseline segmentation accuracy. Post-processing adjustments involve hyperparameters like maximum segment distance for selecting viable segments, field size, and path-finding costs. Iteratively optimizing these hyperparameters is recommended for improved robustness and accuracy. This work also paves the way for joint CRF and CNN training. End-to-end trainable models, such as those that model CRFs and their feature functions as recurrent neural networks, can be appended to CNNs for training [83], which are expected to yield even higher segmentation accuracies [8].
In this study, T-junctions—connections between isolated points and segments—are not included, despite constituting about half of the training set’s segment candidates. Their inclusion is expected to improve segmentation performance [79], offering a fuller network representation. Beyond the seven interface features examined in this study, future iterations could integrate additional grain-related features, such as the Mullins grain growth model [48] to inform the algorithm about potential GB movements for enhanced predictive accuracy. Combining electron diffraction with optical reflectance measurements to estimate GB character [64], and incorporating crystallographic orientation data, can provide detailed grain physical properties, aiding in segmentation refinement. Electron diffraction and optical reflectance could enhance GB identification, while crystallographic orientation might help differentiate grain types or phases. Further segmentation enhancement might come from local neighborhood analysis using image features like brightness, color, texture [47], and the intensity gradient along completion segments [20]. When the original image is accessible, contrast differences between grains and texture analyses could be employed between the original and the predicted masks [28]. This can link GB microstructure with material properties, aiding in predicting mechanical, thermal, and electrical behaviors [15].
Implementing this post-processing procedure facilitates high-throughput experimental pipelines with reduced human supervision requirements, while maintaining comparable accuracies. S(TEM) images can be subjected to computer vision algorithms and subsequently analyzed using complementary methods such as electron diffraction and spectroscopies in a continuous pipeline. Fractional probabilities obtained from computer vision outputs can serve as unary terms in pixel-based CRFs, enhancing predictions and enabling confidence level assignments to each segment for targeted experiments [39].
Statistical distribution information, such as average grain size, could be obtained and leveraged to improve segmentation performance. Outlying grains, indicative of abnormal growth, could be flagged for operator review. The post-processing procedure makes such data available to the operator, alongside candidate segments and their associated costs, thus facilitating a human-in-the-loop approach while expediting the process without compromising fidelity. The segment-based CRF approach exhibits strength in detecting closed contours in noisy environments [79], a capability that can be harnessed to identify interfaces even in the presence of other image features like nanoprecipitates or atomic details from S(TEM) without an additional classifier [42].
Furthermore, the general applicability of this framework extends to less-ideal imaging conditions often encountered in experimental settings, such as in-situ microscopy. While the current work utilized high-quality STEM images, the post-processing method is fundamentally designed to correct fragmented masks, which are characteristic of noisy or low-contrast images. Similarly, for thicker samples or alloys exhibiting complex contrasts from features like segregation-induced complexions, the initial CNN segmentation may be challenged. However, the CRF-based refinement, which operates on the geometric properties of the predicted mask, remains robust. It enforces structural priors like continuity and closure, helping to reconstruct the grain boundary network even when local pixel evidence is ambiguous. Future work could enhance this by incorporating image features specifically sensitive to such complex contrast variations into the CRF’s potential functions.
While the presented work focuses on grain boundaries, the core methodology is adaptable to other domains. The strength of the hierarchical CRF lies in its general framework of leveraging perceptual grouping rules to refine fragmented line networks. The utility of thin segmentation masks extends beyond the domain of materials science, finding applications in biomedical imaging for narrow or elongated structures. Areas of application include retinal blood vessel segmentation [34], neuronal structure analysis [33], coronary artery delineation in cardiac MRI [17], tendon segmentation in musculoskeletal imaging [35], lymphatic system studies [41], and angiography for small blood vessels [82]. However, the specific feature functions and learned weights are necessarily domain-specific. Applying this method to other problems, would require retraining on domain-specific ground truths and could benefit from designing features tailored to the unique characteristics of those images. The methodology introduced in this work offers a new perspective and serves as a resource for other researchers, setting the stage for further advancements in segmentation mask optimization for interconnected line networks.
CRediT authorship contribution statement
Doruk Aksoy: Writing – review & editing, Writing – original draft, Software, Methodology, Investigation, Formal analysis, Data curation, Conceptualization. Huolin L. Xin: Writing – review & editing, Writing – original draft, Funding acquisition. Timothy J. Rupert: Writing – review & editing, Writing – original draft, Supervision, Funding acquisition. William J. Bowman: Writing – review & editing, Writing – original draft, Visualization, Supervision, Project administration, Methodology, Investigation, Funding acquisition, Data curation, Conceptualization.
Code availability
The underlying code for this study is available in the HierarchicalCRF repository and can be accessed via this link: https://github.com/dorukaksoy/HierarchicalCRF.
Declaration of competing interest
All authors declare no financial or non-financial competing interests.
Acknowledgements
This research was primarily supported by the National Science Foundation Materials Research Science and Engineering Center program through the UC Irvine Center for Complex and Active Materials (DMR-2011967).
Supplementary data
Supplementary data to this article can be found online at https://doi.org/10.1016/j.matchar.2025.115694.
Data availability
All data generated or analyzed during this study are included in this published article and its supplementary information files. A preprint version of this manuscript is available on arXiv [7].
References
- [1]I. Adlakha, M.A. Tschopp, K.N. Solanki, The role of grain boundary structure and crystal orientation on crack growth asymmetry in aluminum, Mater. Sci. Eng. A 618 (2014) 345–354, https://doi.org/10.1016/j.msea.2014.08.083.
- [2]D. Aksoy, P. Cao, J.R. Terwilliger, J.P. Warren, T.J. Rupert, Enhanced radiation damage tolerance of amorphous interphase and grain boundary complexions in cu-ta, JOM 76 (2024) 2870–2883, https://doi.org/10.1007/s11837-024-06382-z.
- [3]D. Aksoy, R. Dingreville, D.E. Spearot, An embedded-atom method potential parameterized for sulfur-induced embrittlement of nickel, Modelling Simul. Mater. Sci. Eng. 27 (2019) 085016, https://doi.org/10.1088/1361-651X/ab4c48.
- [4]D. Aksoy, R. Dingreville, D.E. Spearot, Spectrum of embrittling potencies and relation to properties of symmetric-tilt grain boundaries, Acta Mater. 205 (2021) 116527, https://doi.org/10.1016/j.actamat.2020.116527.
- [5]D. Aksoy, J. Luo, P. Cao, T.J. Rupert, A machine learning framework for the prediction of grain boundary segregation in chemically complex environments, Modelling Simul. Mater. Sci. Eng. 32 (2024) 065011, https://doi.org/10.1088/1361-651X/ad585f.
- [6]D. Aksoy, M.J. McCarthy, I. Geiger, D. Apelian, H. Hahn, E.J. Lavernia, J. Luo, H. Xin, T.J. Rupert, Chemical order transitions within extended interfacial segregation zones in NbMoTaW, J. Appl. Phys. 132 (2022) 235302, https://doi.org/10.1063/5.0125202.
- [7]D. Aksoy, H.L. Xin, T.J. Rupert, W.J. Bowman, Human perception-inspired grain segmentation refinement using conditional random fields (arXiv:2312.09968). arXiv, 2023, https://doi.org/10.48550/arXiv.2312.09968.
- [8]A. Arnab, S. Zheng, S. Jayasumama, B. Romera-Paredes, M. Larsson, A. Kirillov, B. Savchynskyy, C. Rother, F. Kahl, P.H.S. Torr, Conditional random fields meet deep neural networks for semantic segmentation: combining probabilistic graphical models with deep learning for structured prediction, IEEE Signal Process. Mag. 35 (1) (2018) 37–52, https://doi.org/10.1109/MSP.2017.2762355.
- [9]F. Bastani, S. He, S. Abbar, M. Alizadeh, H. Balakrishnan, S. Chawla, S. Madden, D. DeWitt, RoadTracer: automatic extraction of road networks from aerial images, in: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 4720–4728, https://doi.org/10.1109/CVPR.2018.00496.
- [10]W.J. Bowman, A. Darbal, P.A. Crozier, Linking macroscopic and Nanoscopic ionic conductivity: a Semipime rical framework for characterizing grain boundary conductivity in polycrystalline ceramics, ACS Appl. Mater. Interfaces 12 (1) (2020) 507–517, https://doi.org/10.1021/acsami.9b15933.
- [11]W.J. Bowman, M.N. Kelly, G.S. Rohrer, C.A. Hernandez, P.A. Crozier, Enhanced ionic conductivity in electroceramics by nanoscale enrichment of grain boundaries with high solute concentration, Nanoscale 9 (44) (2017) 17293–17302, https://doi.org/10.1039/C7NR06941C.
- [12]W.J. Bowman, J. Yang, B. Yildiz, The role of doping and microstructure on hydrogen solubility in monoclinic ZrO2: experimental validations of simulated defect chemistry, Acta Mater. 195 (2020) 172–183, https://doi.org/10.1016/j.actamat.2020.04.020.
- [13]W.J. Bowman, J. Zhu, R. Sharma, P.A. Crozier, Electrical conductivity and grain boundary composition of Gd-doped and Gd/Pr co-doped ceria, Solid State Ionics 272 (2015) 9–17, https://doi.org/10.1016/j.ssi.2014.12.006.
- [14]J.E. Burke, D. Turnbull, Recrystallization and grain growth, Prog. Met. Phys. 3 (1952) 220–292, https://doi.org/10.1016/0502-8205(52)90009-9.
- [15]P.R. Cantwell, T. Frolov, T.J. Rupert, A.R. Krause, C.J. Marvel, G.S. Rohrer, J. M. Rickman, M.P. Harmer, Grain boundary complexions, Annu. Rev. Mater. Res. 50 (1) (2020) 465–492, https://doi.org/10.1146/annurev-matsci-081619-114055.
- [16]V. Caselles, R. Kimmel, G. Sapiro, Geodesic active contours, Int. J. Comput. Vis. 22 (1) (1997) 61–79, https://doi.org/10.1023/A:1007979827043.
- [17]C. Chen, C. Qin, H. Qiu, G. Tarroni, J. Duan, W. Bai, D. Rueckert, Deep learning for cardiac image segmentation: a review, Front. Cardiovasc. Med. 7 (2020), https://www.frontiersin.org/articles/10.3389/fvcm.2020.00025.DOI
- [18]L.-C. Chen, G. Papandreou, F. Schroff, H. Adam, Rethinking Atrous convolution for semantic image segmentation, 2017 arXiv, https://arxiv.org/abs/1706.05587v3.
- [19]B. Cheng, G. Kirschick, P. Dollár, A.C. Berg, A. Kirillov, Boundary IoU: Improving Object-Centric Image Segmentation Evaluation, 2021, March 30 arXiv. https://arxiv.org/abs/2103.16562v1.
- [20]Cheuk Yiu Ip, A. Varshney, J. Jaja, Hierarchical exploration of volumes using multilevel segmentation of the intensity-gradient histograms, IEEE Trans. Vis. Comput. Graph. 18 (12) (2012) 2355–2363, https://doi.org/10.1109/TVCG.2012.231.
- [21]M. Coster, X. Arnould, J.L. Chermant, L. Chermant, T. Chartier, The use of image analysis for sintering investigations: the example of CeO2 doped with TiO2, J. Eur. Ceram. Soc. 25 (15) (2005) 3427–3435, https://doi.org/10.1016/j.jeurceramsoc.2004.09.003.
- [22]A. Delong, A. Osokin, H.N. Isack, Y. Boykov, Fast approximate energy minimization with label costs, in: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2010, pp. 2173–2180, https://doi.org/10.1109/CVPR.2010.5539897.
- [23]O. Dengiz, A.E. Smith, I. Nettleship, Grain boundary detection in microstructure images using computational intelligence, Comput. Ind. 56 (8–9) (2005) 854–866, https://doi.org/10.1016/j.compind.2005.05.012.
- [24]R. D. Dufour, D. Aksø y, D.E. Spearot, A primer on selecting grain boundary sets for comparison of interfacial fracture properties in molecular dynamics simulations, Sci. Rep. 7 (2017) 8332, https://doi.org/10.1038/s41598-017-08637-z.
- [25]I. Duluac, C. Helmer, C. Delcourt, M. Beurton-Aimar, Ensuring a connected structure for retinal vessels deep-learning segmentation, in: 2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), 2023, pp. 2356–2365, https://doi.org/10.1109/ICCVW60793.2023.00250.
- [26]P. Garg, Z. Pan, V. Turlo, T.J. Rupert, Segregation competition and complexion coexistence within a polycrystalline grain boundary network, Acta Mater. 218 (2021) 117213, https://doi.org/10.1016/j.actamat.2021.117213.
- [27]A.P. Garland, B.C. White, B.H. Jared, M. Heiden, D. Donahue, B.L. Boyce, Deep convolutional neural networks as a rapid screening tool for complex additively manufactured structures, Add. Manuf. 35 (2020) 101217, https://doi.org/10.1016/j.addma.2020.101217.
- [28]M. Gauch, Image segmentation and analysis via multiscale gradient watershed hierarchies, IEEE Trans. Image Process. 8 (1) (1999) 69–79, https://doi.org/10.1109/83.736688.
- [29]P.V. Gorsevski, C.M. Onasch, J.R. Farver, X. Ye, Detecting grain boundaries in deformed rocks using a cellular automata approach, Comput. Geosci. 42 (2012) 136–142, https://doi.org/10.1016/j.cageo.2011.09.008.
- [30]K. He, G. Gkioxari, P. Dollár, R. Girshick, Mask R-CNN, arXiv, https://arxiv.org/abs/1703.06870v3, 2017.
- [31]K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, arXiv, 2015, https://doi.org/10.48550/arXiv.1512.03385.
- [32]W.E. Howden, The sofa problem, Comput. J. 11 (3) (1968) 299–301, https://doi.org/10.1093/comjnl/11.3.299.
- [33]H. Ikeno, A. Kumaraswamy, K. Kai, T. Wachtler, H. Ai, A segmentation scheme for complex neuronal Arbors and application to vibration sensitive neurons in the honeybee brain, Front. Neuroinform. 12 (2018), https://www.frontiersin.org/articles/10.3389/fninf.2018.00061.DOI
- [34]S. Jahangard, M.H. Zangooei, M. Shahedi, U-Net Based Architecture for an Improved Multiresolution Segmentation in Medical Images, arXiv, 2020, https://doi.org/10.48550/ARXIV.2007.08238.
- [35]E. Keles, I. Irmakci, U. Bagci, Musculoskeletal MR image segmentation with artificial intelligence, Adv. Clin. Radiol. 4 (1) (2022) 179–188, https://doi.org/10.1016/j.yacr.2022.04.010.
- [36]D.P. Kingma, J. Ba, Adam: A method for stochastic optimization (arXiv:1412.6980), arXiv, 2017, https://doi.org/10.48550/arXiv.1412.6980.
- [37]S.-T. Ko, T. Lee, J. Qi, D. Zhang, W.-T. Peng, X. Wang, W.-C. Tsai, S. Sun, Z. Wang, W.J. Bowman, S.P. Ong, X. Pan, J. Luo, Compositionally complex perovskite oxides: discovering a new class of solid electrolytes with interface-enabled conductivity improvements, Matter 6 (7) (2023) 2395–2418, https://doi.org/10.1016/j.matt.2023.05.035.
- [38]P. Krähenbühl, V. Koltun, Efficient inference in fully connected CRFs with Gaussian edge potentials, 2012 arXiv, http://arxiv.org/abs/1210.5644.
- [39]C. Kunselman, S. Sheikh, M. Mikkelsen, V. Attari, R. Arroyoave, Microstructure classification in the unsupervised context, Acta Mater. 223 (2022) 117434, https://doi.org/10.1016/j.actamat.2021.117434.
- [40]R. KV, K. Prasad, P. Peralam Yegneswaran, Segmentation and classification approaches of clinically relevant curvilinear structures: a review, J. Med. Syst. 47 (1) (2023) 40, https://doi.org/10.1007/s10916-023-01927-2.
- [41]P.-Y. Lai, C.-H. Chang, H.-R. Su, W.-C. Kuo, Lymphatic vessel segmentation in optical coherence tomography by adding U-net-based CNN for artifact minimization, Biomed. Opt. Express 11 (5) (2020) 2679–2693, https://doi.org/10.1364/BOE.389373.
- [42]M. Li, D. Chen, S. Liu, F. Liu, Grain boundary detection and second phase segmentation based on multi-task learning and generative adversarial network, Measurement 162 (2020) 107857, https://doi.org/10.1016/j.measurement.2020.107857.
- [43]S.F. Li, J.K. Mason, J. Lind, M. Kumar, Quadruple nodes and grain boundary connectivity in three dimensions, Acta Mater. 64 (2014) 220–230, https://doi.org/10.1016/j.actamat.2013.10.033.
- [44]A. Liu, W. Lin, M. Narwaria, Image quality assessment based on gradient similarity, IEEE Trans. Image Process. 21 (4) (2012) 1500–1512, https://doi.org/10.1109/TIP.2011.2175935.
- [45]X. Liu, J. Zhang, Y. Jin, S. Bi, M. Eisenbach, Y. Wang, Monte Carlo simulation of order-disorder transition in refractory high entropy alloys: a data-driven approach, Comput. Mater. Sci. 187 (2021) 110135, https://doi.org/10.1016/j.commatsci.2020.110135.
- [46]J. Long, E. Shelhamer, T. Darrell, Fully convolutional networks for semantic segmentation, in: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 3431–3440, https://doi.org/10.1109/CVPR.2015.7298965.
- [47]D.R. Martin, C.C. Fowlkes, J. Malik, Learning to detect natural image boundaries using local brightness, color, and texture cues, IEEE Trans. Pattern Anal. Mach. Intell. 26 (5) (2004) 530–549, https://doi.org/10.1109/TPAMI.2004.1273918.
- [48]W.W. Mullins, Two-dimensional motion of idealized grain boundaries, J. Appl. Phys. 27 (8) (2004) 900–904, https://doi.org/10.1063/1.1722511.
- [49]S. Najmaei, Z. Liu, W. Zhou, X. Zou, G. Shi, S. Lei, B.I. Yakobson, J.-C. Idrobo, P. M. Ajayan, J. Lou, Vapour phase growth and grain boundary structure of molybdenum disulfide atomic layers, Nat. Mater. 12 (8) (2013), https://doi.org/10.1038/nmat3673.
- [50]G. Papari, N. Petkov, Edge and line oriented contour detection: state of the art, Image Vis. Comput. 29 (2) (2011) 79–103, https://doi.org/10.1016/j.imavis.2010.08.009.
- [51]S. Patala, J.K. Mason, C.A. Schuh, Improved representations of misorientation information for grain boundary science and engineering, Prog. Mater. Sci. 57 (8) (2012) 1383–1425, https://doi.org/10.1016/j.pmatsci.2012.04.002.
- [52]M.J. Patrick, J.K. Eckstein, J.R. Lopez, S. Tod eras, S.A. Asher, S.I. Whang, S. Levine, J.M. Rickman, K. Barmak, Automated grain boundary detection for bright-field transmission electron microscopy images via U-net, Microsc. Microanal. 29 (6) (2023) 1968–1979, https://doi.org/10.1093/micmic/ozad115.
- [53]S.J. Pennycook, Z-contrast stem for materials science, Ultramicroscopy 30 (1) (1989) 58–69, https://doi.org/10.1016/0304-3991(89)90173-3.
- [54]H. Peregrina-Barreto, I.R. Terol-Villalobos, J.J. Rangel-Magdaleno, A.M. Herrera-Navarro, L.A. Morales-Hernández, F. Manríquez-Guerrero, Automatic grain size determination in microstructures using image processing, Measurement 46 (1) (2013) 249–258, https://doi.org/10.1016/j.measurement.2012.06.012.
- [55]R. Perera, D. Guzzetti, V. Agrawal, Optimized and autonomous machine learning framework for characterizing pores, particles, grains and grain boundaries in microstructural images, Comput. Mater. Sci. 196 (2021) 110524, https://doi.org/10.1016/j.commatsci.2021.110524.
- [56]P.J. Phillips, M.C. Brandes, M. Mills, M. De Graef, Diffraction contrast STEM of dislocations: imaging and simulations, Ultramicroscopy 111 (9) (2011) 1483–1487, https://doi.org/10.1016/j.ultramic.2011.07.001.
- [57]K.V. Price, Differential evolution, in: I. Zelinka, V. Snášel, A. Abraham (Eds.), Handbook of Optimization: From Classical to Modern Approach, Springer, 2013, pp. 187–214, https://doi.org/10.1007/978-3-642-30504-7_8.
- [58]M. Ragone, M.T. Saray, L. Long, R. Shahbazian-Yassar, F. Mashayek, V. Yurkov, Deep learning for mapping element distribution of high-entropy alloy in scanning transmission electron microscopy images, Comput. Mater. Sci. 201 (2022) 110905, https://doi.org/10.1016/j.commatsci.2021.110905.
- [59]M. Rezanejad, S. Gupta, C. Gummaluru, R. Marten, J. Wilder, M. Gruninger, D. B. Walther, Contour-guided image completion with perceptual grouping (arXiv: 2111.11322). arXiv, 2021, https://doi.org/10.48550/arXiv.2111.11322.
- [60]O. Ronneberger, P. Fischer, T. Brox, U-Net: Convolutional Networks for Biomedical Image Segmentation, 2015, May 18 arXiv.Org. https://arxiv.org/abs/1505.04597v1.
- [61]K. Roy Choudhury, P.A. Meere, K.F. Mulchrone, Automated grain boundary detection by CASRG, J. Struct. Geol. 28 (3) (2006) 363–375, https://doi.org/10.1016/j.jsg.2005.12.010.
- [62]N. Salman, Image segmentation based on watershed and edge detection techniques, Int. Arab J. Inf. Technol. 3 (2006) 104–110.
- [63]D. Scheiber, L. Románer, Impact of the segregation energy spectrum on the enthalpy and entropy of segregation, Acta Mater. 221 (2021) 117393, https://doi.org/10.1016/j.actamat.2021.117393.
- [64]M. Seita, M. Volpi, S. Patala, I. McCue, C.A. Schuh, M.V. Diamanti, J. Erlebacher, M.J. Demkowicz, A high-throughput technique for determining grain boundary character non-destructively in microstructures with through-thickness grains, npj Comput. Mater. 2 (1) (2016), https://doi.org/10.1038/npjcompumats.2016.16. Article 1.
- [65]J. Shotton, J. Winn, C. Rother, A. Criminisi, TextonBoost for image understanding: multi-class object recognition and segmentation by jointly Modeling texture, layout, and context, Int. J. Comput. Vis. 81 (1) (2009) 2–23, https://doi.org/10.1007/s11263-007-0109-1.
- [66]K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition, arXiv, 2015, https://doi.org/10.48550/arXiv.1409.1556.
- [67]C. Sutton, An introduction to conditional random fields, Found. Trends Mach. Learn. 4 (4) (2012) 267–373, https://doi.org/10.1561/2200000013.
- [68]K. Syed, N.B. Motley, W.J. Bowman, Heterointerface and grain boundary energies, and their influence on microstructure in multiphase ceramics, Acta Mater. 227 (2022) 117685, https://doi.org/10.1016/j.actamat.2022.117685.
- [69]K. Syed, M. Xu, K.K. Ohtaki, D. Kok, K.K. Karandikar, O.A. Graeve, W.J. Bowman, M.L. Mecartney, Correlations of grain boundary segregation to sintering techniques in a three-phase ceramic, Materialia 14 (2020) 100890, https://doi.org/10.1016/j.mtla.2020.100890.
- [70]X. Tong, W.J. Bowman, A. Mejia-Giraldo, P.A. Crozier, D.S. Mebane, New data-driven interacting-defect model describing Nanoscopic grain boundary compositions in ceramics, J. Phys. Chem. C 124 (43) (2020) 23619–23625, https://doi.org/10.1021/acs.jpcc.0c05713.
- [71]K.P. Treder, C. Huang, J.S. Kim, A.I. Kirkland, Applications of deep learning in electron microscopy, Microscopy 71 (Supplement_1) (2022) i100–i115, https://doi.org/10.1093/micro/dfac043.
- [72]H. Vahidi, K. Syed, H. Guo, X. Wang, J.L. Wardini, J. Martinez, W.J. Bowman, A review of grain boundary and Heterointerface characterization in polycrystalline oxides (by scanning) transmission electron microscopy, Crystals 11 (8) (2021) 878, https://doi.org/10.3390/cryst11080878.
- [73]M. Vimal, S. Sandfeld, A. Prakash, Grain segmentation in atomistic simulations using orientation-based iterative self-organizing data analysis, Materialia 21 (2022) 101314, https://doi.org/10.1016/j.mtla.2022.101314.
- [74]M. Wagih, C.A. Schuh, Viewpoint: can symmetric tilt grain boundaries represent polycrystals? Scr. Mater. 237 (2023) 115716 https://doi.org/10.1016/j.scriptamat.2023.115716.
- [75]J.L. Wardini, H. Vahidi, H. Guo, W.J. Bowman, Probing multiscale disorder in Pyrochlore and related complex oxides in the transmission electron microscope: a review, Front. Chem. 9 (2021) 743025, https://doi.org/10.3389/fchem.2021.743025.
- [76]L.R. Williams, D.W. Jacobs, Stochastic completion fields: a neural model of illusory contour shape and salience, Neural Comput. 9 (4) (1997) 837–858, https://doi.org/10.1162/neco.1997.9.4.837.
- [77]G.J. Woezinger, Exact algorithms for NP-hard problems. A survey, in: M. Jünger, G. Reinelt, G. Rinaldi (Eds.), Combinatorial Optimization—Eureka, you Shrink! Papers Dedicated to Jack Edmonds 5th International Workshop Aussois, France, March 5–9, 2001 Revised Papers, Springer, 2003, pp. 185–207, https://doi.org/10.1007/3-540-36478-1_17.
- [78]P. Wynblatt, D. Chatain, Anisotropy of segregation at grain boundaries and surfaces, Metall. Mater. Trans. A 37 (9) (2006) 2595–2620, https://doi.org/10.1007/BF02586096.
- [79]Yansheng Ming, Hongdong Li, Xuming He, Connected contours: A new contour completion model that respects the closure effect, in: 2012 IEEE Conference on Computer Vision and Pattern Recognition, 2012, pp. 829–836, https://doi.org/10.1109/CVPR.2012.6247755.
- [80]W. Yin, D. Brittain, J. Borseth, M.E. Scott, D. Williams, J. Perkins, C.S. Own, M. Murfitt, R.M. Torres, D. Kapner, G. Mahalingam, A. Bleckert, D. Castelli, D. Reid, W.-C.A. Lee, B.J. Graham, M. Takeno, D.J. Bumbarger, C. Farrell, R. C. Reid, N.M. da Costa, A petascale automated imaging pipeline for mapping neuronal circuits with high-throughput transmission electron microscopy, Nat. Commun. 11 (1) (2020), https://doi.org/10.1038/s41467-020-18659-3. Article 1.
- [81]K. Zhang, H.D. Cheng, B. Zhang, Unified approach to pavement crack and sealed crack detection using Reclassification based on transfer learning, J. Comput. Civ. Eng. 32 (2) (2018) 04018001, https://doi.org/10.1061/(ASCE)CP.1943-5487.0000736.
- [82]M. Zhang, Z. Chang, X. Wu, X. Cao, G.S. Young, H. Chen, X. Xu, A neural network approach to segment brain blood vessels in digital subtraction angiography, Comput. Methods Prog. Biomed. 185 (2020) 105159, https://doi.org/10.1016/j.cmpb.2019.105159.
- [83]S. Zheng, J. Jayasumana, B. Romera-Paredes, V. Vineet, Z. Su, D. Du, C. Huang, P. H.S. Torr, Conditional random fields as recurrent neural networks, in: 2015 IEEE International Conference on Computer Vision (ICCV), 2015, pp. 1529–1537, https://doi.org/10.1109/ICCV.2015.179.