betaFeedback

WorksComputer Science

GRATEV2.0: Computational Tools for Real-time Analysis of High-throughput High-resolution TEM (HRTEM) Images of Conjugated Polymers

Abstract

Automated analysis of high-resolution transmission electron microscopy (HRTEM) images is increasingly essential for advancing research in organic electronics, where precise characterization of nanoscale crystal structures is crucial for optimizing material properties. This paper introduces an open-source computational framework called GRATEV2.0 (GRaph-based Analysis of TEM), designed for real-time analysis of HRTEM data, with a focus on characterizing complex microstructures in conjugated polymers, illustrated using Poly[N-9'-heptadecanyl-2,7-carbazole-alt-5,5-(4',7'-di-2-thienyl-2',1',3'-benzothiadiazole)] (PCDTBT), a key material in organic photovoltaics. GRATEV2.0 employs fast, automated image processing algorithms, enabling rapid extraction of structural features like d-spacing, orientation, and crystal shape metrics. Gaussian process optimization rapidly identifies the user-defined parameters in the approach, reducing the need for manual parameter tuning and thus enhancing reproducibility and usability. Additionally, GRATEV2.0 is compatible with high-performance computing (HPC) environments, allowing for efficient, large-scale data processing at near real-time speeds. A unique feature of GRATEV2.0 is a Wasserstein distance-based stopping criterion, which optimizes data collection by determining when further sampling no longer adds statistically significant information. This capability optimizes the amount of time the TEM facility is used while ensuring data adequacy for in-depth analysis. Open-source and tested on a substantial PCDTBT dataset, this tool offers a powerful, robust, and accessible solution for high-throughput material characterization in organic electronics.

Introduction

Microscopy has long been a cornerstone in materials science, offering a unique window into the microstructure and enabling scientists to study properties at various scales, from the millimeter down to the atomic level [1, 2]. By visualizing otherwise inaccessible structures, microscopy provides critical insights into how atomic and molecular arrangements influence key macroscopic properties such as mechanical strength, electrical conductivity, and chemical reactivity [3]. This fundamental understanding is essential for designing advanced materials with tailored properties for applications in fields such as electronics, energy storage, and catalysis.

High-resolution transmission electron microscopy (HRTEM) has advanced significantly in recent years, transforming nanoscale imaging and allowing researchers to capture atomic-level details of materials [4]. HRTEM can now achieve sub-angstrom spatial resolution, making it possible to directly observe atomic lattices, defects, and interfaces that govern a material’s behavior [5]. The development of automated data acquisition systems has further broadened HRTEM’s capabilities, enabling the collection of extensive datasets comprising hundreds or even thousands of high-resolution images [6]. These technological advances have opened new avenues for studying complex materials, such as organic semiconductors and conjugated polymers, with applications in organic electronics and photovoltaics [7].

While modern HRTEM has enhanced our ability to study complex materials, it also introduces challenges in data management and analysis due to the sheer volume and complexity of the data generated [8]. The scale of high-resolution datasets can quickly overwhelm traditional analysis workflows, which are often manual and time-consuming. This manual approach is highly dependent on the expertise and subjective judgment of the experimentalist, making it challenging to ensure consistency and reproducibility. In high-throughput applications where rapid feedback is essential, such as optimizing synthesis conditions or tracking real-time structural changes, the limitations of manual analysis become particularly pronounced.

To address these challenges, automated methods for HRTEM data analysis have emerged over the past decade. These methods aim to extract quantitative structural information from digital micrographs with minimal human intervention, improving both the efficiency and reliability of analysis [3, 9, 10, 11, 12]. Automated approaches can be broadly categorized into offline (post-acquisition) and online (real-time) methods. Offline methods involve processing data after acquisition, which is useful for extracting detailed structural information but may be computationally intensive and unable to keep pace with data acquisition rates [9, 11, 12, 13]. A motivating example is our prior work – GRATE (GRaph based Analysis of TEM images) [13] – that is based on Matlab. The computational and memory overhead makes offline methods less suitable for high-throughput applications where rapid processing is needed. The reliance on post-acquisition processing has underscored the need for integrating HRTEM with high-performance computing (HPC) resources to handle large volumes of data efficiently.

In recent years, there has been a strong push toward developing in situ, or real-time, automated analysis methods [14, 15, 16]. In situ analysis allows for data interpretation during experiments, providing immediate feedback that can guide adjustments in experimental parameters. This capability is particularly valuable in dynamic experiments, such as observing structural changes in response to external stimuli or monitoring materials during synthesis [17, 18, 19, 20, 21, 22, 23]. Real-time analysis requires methods that can handle high-resolution data quickly and accurately, maintaining performance under the demanding conditions of live data acquisition.

Machine learning (ML) has shown promise in automating feature recognition in HRTEM images, with deep learning models capable of identifying patterns and structures in high-resolution data [24, 25, 26]. However, ML approaches have limitations in this context. They typically require large, annotated datasets for training, which is resource-intensive and time-consuming to create [25, 27, 28]. Each chemical system may exhibit unique structural features and imaging artifacts, requiring retraining or fine-tuning of models to ensure accuracy [9, 12]. This dependence on extensive training data reduces the adaptability of ML methods across different material systems, particularly in high-throughput or exploratory research where materials and imaging conditions vary frequently. Recent work in creating foundational models, and machine learning approaches like transfer learning and few-shot learning applied to these problems show promise.

In contrast, robust image processing techniques provide a more flexible and efficient solution for automated analysis. Image processing-based approaches can generate interpretable, reproducible data in real time without the need for extensive, system-specific training. These methods typically involve well-established procedures like frequency filtering, binarization, and morphological operations that are adaptable across diverse materials [9, 11, 13, 29, 30, 31]. In conjunction with ML models, image processing methods offer a practical solution for rapid, in situ data analysis with minimal human input, making them particularly valuable for high-throughput studies in materials science.

GRATEv2 aims to bridge the gap between offline and real-time analysis in high-resolution transmission electron microscopy (HRTEM) by providing an automated, image-processing-based framework with minimal human intervention. Tailored for high-throughput settings, GRATEv2 combines rapid data extraction with a robust and user-friendly parameter optimization process. By focusing on image processing techniques augmented with Gaussian process optimization, GRATEv2 minimizes the need for manual parameter selection and tuning, enhancing reproducibility and accessibility for researchers.

A schematic overview of the GRATEv2 computational framework is shown in Figure 1a. The framework processes raw HRTEM images of PCDTBT (Figure 2a), applies preprocessing, and performs automated image processing with parameters optimized via Bayesian optimization (Figure 1b). Traditionally, manual annotation of crystals in HRTEM images is time-consuming and subjective (Figure 2). In our approach, only a dozen manually annotated images (Figure 2b) are used as input to a Bayesian optimizer to rapidly identify material-specific image processing parameters. This minimal requirement significantly reduces the burden on researchers, facilitating rapid deployment of the algorithm on new datasets.

Image not extracted

Figure 1. Schematic overview of the GRATEv2 computational framework. The framework processes raw HRTEM images of PCDTBT, applies preprocessing, and performs automated image processing with parameters optimized via Bayesian optimization. A data sufficiency criterion based on the Wasserstein distance assesses whether additional TEM data is needed. The output comprises extracted structural features such as d-spacing, orientation, and crystal shape metrics. (a) The overall computational framework of GRATEv2, and (b) the detailed Bayesian Optimization (BO) component used for parameter tuning.

Comparison between (a) the original HRTEM image and (b) the manually annotated crystals: (a) Algorithm input; (b) Hand annotated crystal

Figure 2. Comparison between (a) the original HRTEM image and (b) the manually annotated crystals used for training the Bayesian model. 13 manually annotated images are used for effective training in the work. Manual annotation is time-consuming and subjective.

GRATEv2 introduces several key innovations to address the limitations of current HRTEM analysis methods:

  1. GRATEv2 offers fast and efficient processing of HRTEM images on the order of a few seconds per image. By supporting batch processing and multiprocessing, it enables rapid analysis of large datasets, significantly improving throughput.

  1. It extends advanced image analysis techniques to organic polymer materials, specifically PCDTBT, which have not been extensively studied with these methods.

  1. Bayesian optimization is employed within GRATEv2 to automate the tuning of material-specific image processing parameters, allowing the framework to adapt to different datasets with minimal expert input (Figure 1b).

  1. The algorithm parameters are constructed as functions of known dd-spacing values, simplifying parameter selection and making the method more accessible to users without extensive image processing expertise. This also ensures that parameter selection is interpretable and, thus, scientifically justified.

  1. GRATEv2 incorporates a data sufficiency criterion based on the Wasserstein distance to guide data collection efforts. This criterion provides a quantitative stopping point, indicating when further TEM data collection no longer yields additional insights, which is important when access to imaging facilities is limited and imaging is expensive.

By optimizing data collection, GRATEv2 helps experimentalists avoid unnecessary resource expenditure while ensuring data quality. This combination of fast, automated processing, real-time adaptability, and efficient data collection positions GRATEv2 as a powerful tool for materials research, particularly in the study of organic electronic materials such as conjugated polymers [32, 33, 34].

Results

We first provide brief details of the material and process used to collect the HRTEM data for completeness. To prepare TEM samples, 5 mg/mL solutions of PCDTBT were dissolved in chlorobenzene within a nitrogen glovebox at 45°C for at least 12 hours. Silicon wafers were cleaned by sonication for 20 minutes in acetone, followed by 20 minutes in isopropanol, and subsequently subjected to UV-ozonation for 20 minutes. Poly(3,4-ethylenedioxythiophene)-poly(styrene sulfonate) (PEDOT:PSS) films were cast onto silicon substrates by spin coating at 4000 RPM for 2 minutes in air to serve as a sacrificial layer, facilitating film floating. Substrates were then transferred to a nitrogen glovebox, where PCDTBT films were spin-cast at 800 RPM for 2 minutes. For TEM sample preparation, the coated substrates were removed from the glovebox, floated in deionized water, and carefully transferred onto copper TEM grids. These samples were left under ambient conditions to dry overnight and then annealed in the nitrogen glovebox at 190 °C for 2 hours. High-resolution TEM (HRTEM) imaging was performed at the Penn State Materials Characterization Lab using the FEI Titan Krios microscope operating at 300 kV, equipped with the K2 direct electron detector and a cryo-stage. A dose rate of 20 e/A˚2s20\,\mathrm{e}/\text{\AA}^{2}\mathrm{s} was used for a 2.5 s exposure. An automated acquisition process was set to autofocus before each capture at 300 kx magnification, with a randomly assigned defocus value between 0 and −3 μm-3\,\mu\mathrm{m}. Images were acquired at 470 kx magnification with a 2.5 μm2.5\,\mu\mathrm{m} step size between exposed regions, resulting in a total of 637 images for subsequent analysis.

Using GRATEv2, we detected crystals within each of the HRTEM images. For each identified crystal, the following features were extracted (see § 2.2): center-of-mass coordinates, orientation angle relative to the image axis, d-spacing, and crystal lengths along both the major and minor axes. Through this process, GRATEv2 identified a total of 4350 ordered domains from the HRTEM images. In § 2.3, we present the timing performance of GRATEv2, along with the time distribution among different components of the analysis process. In § 2.4, we evaluate the data sufficiency—that is, how many images are sufficient to achieve statistical convergence in the extracted features.

Advantages of Bayesian Parameter Optimization

To evaluate the effectiveness of integrating Bayesian optimization for hyperparameter tuning in our image processing algorithm, we conducted a comparative analysis between manually selected parameters and the Bayesian-optimized parameters. The performance was assessed using the Intersection over Union (IoU) metric between the algorithm’s output against expert-annotated ground truth on six representative HRTEM images.

Quantitative Comparison of IoU Scores Table 1 presents the IoU scores for each image using both the manually selected parameters and the Bayesian-optimized parameters. Additionally, the table includes the differences in IoU scores between the two parameter sets for each image. The average and standard deviation of IoU score across the six images is also calculated for both cases.

Image FilenameManual Selected Parameters IoUBayesian Parameters IoUIoU Difference, did_i
1.tif0.52960.6911+0.1615
2.tif0.32740.5156+0.1882
3.tif0.31780.4934+0.1756
4.tif0.52850.5864+0.0579
5.tif0.39060.5400+0.1494
6.tif0.52880.6438+0.1150
Average IoU0.43710.5784+0.1413

Table 1. IoU Scores over validation dataset for Manual and Bayesian-Optimized Parameters with Differences

The average IoU score using the Bayesian-optimized parameters is 0.5784, representing an improvement of approximately 32.3% over the average IoU of 0.4371 obtained with the manually selected parameters. This significant increase demonstrates the effectiveness of Bayesian optimization in enhancing the algorithm’s performance in detecting and segmenting crystalline regions in HRTEM images. The computed tt-value corresponding to tt-statistic is 7.074, which exceeds the critical tt-value of 2.571, indicating strong statistical justification for improvement.

Visual Comparison of Detection Results Figure 3 and Appendix § 12 provide qualitative comparisons between the segmentation results obtained with manually selected parameters and those obtained with Bayesian-optimized parameters. Visual comparisons illustrate that the algorithm with Bayesian-optimized parameters generally provides a more accurate and comprehensive detection of crystalline regions than the manually tuned approach. The Bayesian-optimized parameters tend to align the detected crystal boundaries more consistently with the annotated features. In particular, the algorithm with Bayesian-optimized parameters appears to capture finer details and produce clearer delineation of crystal boundaries, suggesting a closer correspondence to the expert annotations than that achieved under manual parameter tuning.

Image not extracted

Figure 3. Figure 3: Comparison of Ground Truth, Manually selected parameter detection, and Bayesian-Optimized parameter detection across three different images. Each column represents a distinct detection method, illustrating how Bayesian optimization enhances segmentation accuracy by more closely matching the ground truth annotations compared to manual parameter tuning. The Bayesian optimization process was conducted over 200 iterations, achieving a minimum loss value of -0.7319 at the 151st evaluation.

Convergence of the Bayesian Optimization Process Figure 4 depicts the convergence of the Bayesian optimization process. The objective function, defined as the negative mean IoU, decreases over successive iterations, indicating that the optimization algorithm effectively identifies hyperparameters that enhance the segmentation performance. The convergence plot demonstrates that significant improvements are achieved within the first 51 iterations, after which the objective function gradually approaches a plateau.

Image not extracted

Figure 4. Convergence of the Bayesian optimization process over 200 Iterations, illustrating the reduction in the loss function (negative IoU) as optimization progresses. The y-axis represents the minimum loss value achieved up to that evaluation. The minimum loss value of -0.7319 was attained at the 151st evaluation.

Analysis of Optimized Parameters Appendix § 10 compares the values of the manually selected parameters with those obtained through Bayesian optimization.

Several notable differences are observed between the parameter sets:

  • Morphological Operations: The Bayesian-optimized parameters for closing_k_size and opening_k_size are significantly smaller than the manual values (2 vs. 15 and 17, respectively). This suggests that less aggressive morphological operations preserve finer details in the images, contributing to improved segmentation accuracy.

  • Edge Detection and Filtering: Parameters like ellipse_len_propCons and dspace_bandpass are adjusted to better capture the characteristics of the crystalline structures. The increase in ellipse_len_propCons from 1.5 to 4.03 indicates a preference for detecting longer ellipses, aligning with the elongated shapes of crystals.

  • Thresholding Parameters: The pixThresh_propCons is slightly higher in the Bayesian-optimized parameters, which may help in differentiating crystals from the background noise more effectively.

Advantages of Bayesian Optimization The integration of Bayesian optimization into our image processing framework offers several advantages:

  1. Enhanced Performance: The Bayesian-optimized parameters significantly improve the mean IoU score by approximately 32.3%, indicating superior segmentation performance.

  2. Automated Tuning: Bayesian optimization automates the hyperparameter tuning process, reducing the need for manual intervention and deep image-processing expertise, thereby saving time and resources. This is especially important when the number of hyperparameters (here, 13) makes manual exploration rather tedious.

  3. Efficient Exploration: The optimization process efficiently explores the hyperparameter space, converging to optimal values in fewer evaluations compared to exhaustive search methods. This is evident from the convergence plot in Figure 4.

  4. Robustness: The optimized parameters produce robust results across various images, as evidenced by consistent improvements in IoU scores.

Structural Feature Extraction of Crystals from HRTEM Images

Using the parameter set listed in Appendix § 10, we applied GRATEv2 to detect and analyze crystalline domains within the HRTEM images of PCDTBT. The segmentation results for two images are presented in Figure 5, where the identified crystals are highlighted. Each detected crystal is surrounded by a convex hull boundary, with a shaded region representing a more precise delineation of the crystallite. A line at the crystal centroid indicates its orientation. The extracted properties for these detected crystals are summarized in Table 2, including their centroid coordinates, area, orientation angle, d-spacing, major and minor axis lengths, and axis angle. Additionally, the correlation measurements between pairs of crystals are provided in Table 3, featuring metrics such as metric distance, direct distance, and relative angle.

NameCentroid (px,pxpx, px)Area (nm2nm^2)Angle (degdeg)d-Spacing (nmnm)MajorAxis (nmnm)MinorAxis (nmnm)AxisAngle (degdeg)
1.tif(1748, 670)589.7-164.72.121.110.223.8
(785, 1992)293.9-55.92.014.89.9-65.5
2.tif(534, 497)177.9-137.91.912.55.353.0
(1607, 402)84.3-136.00.87.94.335.4
(3345, 546)71.7-148.81.714.44.839.6
(2050, 1975)125.4-146.02.216.04.234.1
(1922, 3396)263.8-141.42.014.28.046.9

Table 2. Features of the detected crystals shown in Figure 5.

NameMetric Distance (1)Direct Distance (nmnm)Relative Angle (degdeg)
1.tif0.8920.8471.24
2.tif2.9135.8110.95
1.9526.978.13
2.4540.943.5
2.2124.572.81
2.9140.587.45
1.1718.184.64

Table 3. Crystal correlation measurements for Figure 5.

Image not extracted

Figure 5. (a) and (b) are 1.tif and 2.tif images respectively corresponding to Table 2 and Table 3. (a) and (b) shows the original image (left) and the segmentation output (right) from our algorithm for HRTEM of PCDTBT. The detected crystals have a d-spacing of 1.9nm. The image on the left is the input to the algorithm, and on the right is the output of the algorithm. Each detected crystal in the output shows (1) the convex hull boundary around the crystal, (2) the shaded region representing a more exact crystal region, and (3) a straight line at the centroid of the convex hull, which shows the orientation of the crystal patterns.

The detection of these crystalline domains allows for a comprehensive analysis of the microstructural features of PCDTBT. The d-spacing values shown in Figure 6b range from approximately 1.1nm to 2.9nm, which are consistent with the expected lattice spacings associated with PCDTBT’s crystalline structures [35]. Variations in d-spacing among the detected crystals may be attributed to differences in crystallite orientation, strain within the material, or inherent structural disorder due to the semi-crystalline nature of PCDTBT.

Figure 6. Individual crystal analysis. The properties of the individually detected crystals are plotted: (a) Crystal area, (b) FFT evaluated dd-spacing, (c) Angle difference between crystal’s pattern and its major axis, (d) Crystal aspect ratio, and (e) crystal axis lengths. (a) to (d) are histograms with kernel density estimate, and (e) is a scatter plot from analysis of the entire dataset.

The areas of the detected crystals vary significantly as shown in Figure 6a, with values ranging from approximately 14.89 nm214.89\ \mathrm{nm}^2 to 2307.18 nm22307.18\ \mathrm{nm}^2. Larger crystal areas may correlate with improved charge transport properties, as larger crystalline domains can facilitate more efficient charge carrier mobility along the polymer chains [32]. The aspect ratios, derived from the major and minor axis lengths, provide insights into the shapes of the crystals. Higher aspect ratios indicate elongated, rod-like crystals, while lower aspect ratios suggest more equiaxed or spherical shapes. The diversity in crystal shapes and sizes can impact the overall morphology and performance of the polymer in electronic applications.

Figure 6 also presents additional statistical analyses of individual crystal properties across the dataset, including histograms of crystal areas, dd-spacings, orientation angles, and shape descriptors like aspect ratio. These plots allow us to identify prevalent features and distributions within the material. For example, the histogram of dd-spacings may show a peak around a specific value, indicating a dominant crystalline phase or preferred stacking distance within the polymer chains.

The comprehensive analysis provided by GRATEv2 enables us to establish quantitative relationships between microstructural features and potential material performance. By systematically characterizing the size, shape, orientation, and spatial distribution of crystalline domains, we can correlate these features with electronic properties measured in devices. This level of detailed microstructural understanding is essential for guiding the design and processing of conjugated polymers to achieve optimal performance in organic electronic applications.

For a more extensive and detailed exploration of this PCDTBT dataset, including insights into intercrystalline correlations and preferred crystallographic alignments, readers are referred to the work of our collaborators in [36]. Their study leverages automated HRTEM and the GRATEv2 image processing outputs to unravel how neighboring crystals preferentially align along certain lattice directions, likely reflecting underlying liquid crystalline order within the polymer. By combining the analysis presented here with their comprehensive assessment of orientation correlations and lattice parameters, one obtains a richer and more complete understanding of the polymer’s nanoscale structure and its implications for organic electronics.

Timing Statistics

The algorithm was executed on a computer equipped with a 96-core AMD EPYC 9654 CPU @ 3.7 GHz running Linux OS. The total time for processing a single high-resolution transmission electron microscopy (HRTEM) image with 1.9 nm dd-spacing crystals is approximately 6.52 seconds when utilizing a single core. The timing consumption by various parts of the algorithm is presented in Figure 7. The most time-consuming steps are skeletonization (approximately 4.66 seconds), followed by breaking branches and the preprocessing steps (blurring, histogram equalization, and thresholding). Utilizing all 96 cores, we processed an entire dataset of 637 images in 284 seconds (approximately 4 minutes and 44 seconds), reducing the per-image processing time to just 0.44 seconds. This significant improvement demonstrates the scalability and efficiency of our algorithm when parallelized across multiple cores.

Image not extracted

Figure 7. Time taken by each step in the algorithm for the analysis of 1.9 nm dd-spacing crystals in a single image using a single core. The total processing time is approximately 6.52 seconds.

The performance enhancements of our algorithm are attributed to the use of optimized libraries and functions that efficiently handle computationally intensive tasks. Specifically, we employed:

  1. The skeletonize function from skimage.morphology [37] for efficient skeletonization.

  2. OpenCV [38] functions equalizeHist and threshold (cv2.equalizeHist, cv2.threshold) for fast histogram equalization and image thresholding.

  3. Morphological operations such as closing and opening using cv2.morphologyEx from OpenCV.

  4. The skeleton_to_csgraph function from the skan [39] library for converting skeleton images to graph representations.

  5. The label function from skimage.measure [37] for rapid image segmentation.

  1. Sparse graph structures and connected component analysis using csr_matrix and connected_components from scipy.sparse.csgraph [40] for efficient graph construction and evaluation.

  1. Fast Fourier transforms and other numerical operations using NumPy functions such as np.fft [41].

  1. The alphashape library to create shrink-wraps around point clouds.

By utilizing these optimized libraries, we minimized computational overhead and maximized the efficiency of each processing step. The skeletonization step, although still the most time-consuming, benefits significantly from the optimized implementation in skimage [37]. Similarly, the use of sparse matrices and efficient graph algorithms from scipy.sparse.csgraph [40] greatly accelerates the analysis of the skeleton’s connectivity.

Our analysis of the timing statistics reveals that the skeletonization step accounts for approximately 71% of the total processing time for a single image when executed on a single core. This indicates that skeletonization is a major computational bottleneck in the algorithm. However, due to the parallel nature of image processing tasks, distributing the workload across multiple cores significantly mitigates this bottleneck. By processing images concurrently, we effectively utilize the available computational resources, resulting in a substantial reduction in total processing time.

Furthermore, the efficient handling of large data structures, such as sparse matrices in graph construction and connected component analysis, contributes to the algorithm’s scalability. The use of optimized libraries ensures that even computationally intensive tasks are executed as efficiently as possible. This optimization is crucial when dealing with large datasets, as it enables rapid analysis without compromising accuracy or resolution.

The combination of algorithmic efficiency, optimized library functions, and effective parallelization allows our method to achieve high performance in processing and analyzing large volumes of HRTEM images. This capability is essential for applications requiring rapid data analysis and real-time feedback in materials science and related fields.

Data Sufficiency Analysis of PCDTBT Crystals in HRTEM Images

In our analysis of data sufficiency, we examine how the distribution of crystal areas reaches an asymptotic distribution as we incorporate more HRTEM images of PCDTBT. The complete dataset consists of 837 crystals from around 600 images. To evaluate how quickly the underlying distribution of crystal areas converges, we incrementally add data in fixed-size batches and compute the first-order Wasserstein distance W1W_1 between distributions formed by consecutive increments.

Specifically, we consider four different batch sizes at which images (rather, crystals) are added to the dataset: 10, 21, 42, and 84 crystals. For each batch size scenario, we start from 0 crystals and progressively add data in increments equal to the batch size until we reach the full dataset size (of 837 crystals). After each increment, we compute the Wasserstein distance between the new distribution (including i×batchSizei \times\text{batchSize} crystals) and the previous distribution (with (i−1)×batchSize(i - 1) \times\text{batchSize} crystals). To obtain a more stable and reliable statistic, we repeat this evaluation 10 times at each increment, each time randomly sampling the (i−1)(i - 1) batches from ii batches of crystals and then average the resulting distances.

As shown in Figure 8, the histograms become smoother and more stable as the dataset grows, demonstrating that the distribution of crystal areas converges toward a steady form. Meanwhile, Figure 8e shows how the averaged Wasserstein distance between consecutive increments decreases with additional data. Each curve represents a different batch size, revealing that the scale of incremental changes to the distribution depends on how many crystals are added at once.

Image not extracted

Figure 8. Data sufficiency evaluation for different crystal areas.

Larger batch sizes (e.g., 84 crystals) introduce more data at each increment, leading to more pronounced changes in the distribution per step. Once enough large increments have been included, the distribution may show a sudden, relatively large drop in Wasserstein distance, then rapidly converge. In contrast, smaller batch sizes (e.g., 10 crystals) add data more gradually, producing smoother and more frequent updates. Each small increment makes a subtler change to the distribution, resulting in a more gradual and fine-grained trajectory toward convergence.

Because each batch size scenario scales the increments differently, the threshold for deciding when to stop data collection should also be scaled accordingly. For larger batch sizes, even a modest Wasserstein distance value (e.g., around 4 units for the 84-crystal batch) might indicate sufficient convergence, since one large increment can naturally shift the distribution more. For smaller batch sizes, where each increment is gentler (e.g., around 1.5 units difference for the 10-crystal batch), a lower threshold might be more appropriate, reflecting the finer control and resolution over the distribution’s shape. To illustrate this, Table 4 shows the Wasserstein distance between the full dataset and a dataset that is one batch smaller, for each considered batch size scenario. These values provide concrete examples of how batch size influences the scale of change in the distribution.

Batch SizeWasserstein Distance (Full vs. One Batch Less)
844.12
422.83
212.08
101.51

Table 4. Wasserstein distance between the full dataset distribution and the distribution obtained by removing one batch of data from the full set, for various batch sizes.

With this information, an experimentalist might set a higher threshold for larger batch sizes (e.g., around 4 units for an 84-crystal batch) and a lower threshold for smaller batch sizes (e.g., closer to 1.5 units for a 10-crystal batch). This ensures that the threshold for deciding when to cease data collection remains proportionate to the scale of changes induced by each incremental addition of data.

In practical terms, experimentalists can use insights from Figure 8e and Table 4 to tailor their data collection strategy. By selecting an appropriate batch size and a corresponding Wasserstein distance threshold, they can decide when further data provides diminishing returns. If quick feedback and high resolution of distribution changes are desired, a smaller batch size and a lower threshold can be chosen. If time or resources are limited, larger batch sizes and a slightly higher threshold may be more suitable. In either case, once the averaged Wasserstein distance consistently falls below the chosen threshold, the experimentalist can confidently cease data collection, knowing the distribution is sufficiently representative. This approach transforms data sufficiency from a guesswork exercise into a clear, data-driven criterion that guides experimental resource allocation and ensures that the resulting dataset meets the necessary statistical rigor. An experimentalist might:

  • Begin Data Collection in Batches: They determine a batch size (e.g., 10 crystals per batch) and start collecting HRTEM images in increments of that batch size.

  • Periodic Assessment: After each new batch of data, they compute the Wasserstein distance between the current and previous distributions of crystal areas. This computation can be done after every increment, providing immediate feedback on the impact of newly acquired data.

  • Decision Point: If, after a certain number of increments, the averaged Wasserstein distance consistently falls below the established threshold (e.g., 1.5 units), the experimentalist has quantitative evidence that adding more data is unlikely to yield new insights into the crystal area distribution.

  • Stopping Data Collection: With this statistical criterion, the experimentalist can confidently stop collecting further HRTEM images, reallocating their time and resources. This prevents unnecessary prolonged imaging campaigns.

Conclusions

In this work, we have developed and presented GRATEv2, an open-source computational framework for the automated analysis of high-resolution transmission electron microscopy (HRTEM) images, specifically focusing on complex microstructures in conjugated polymers like PCDTBT. By leveraging fast, automated image processing algorithms augmented with Gaussian process optimization, GRATEv2 significantly reduces the need for manual selection of parameters and tuning, enhancing both reproducibility and user accessibility. The integration of a Wasserstein distance-based stopping criterion within GRATEv2 provides a quantitative method for optimizing data collection, ensuring efficient use of transmission electron microscopy (TEM) resources without compromising data quality.

GRATEv2’s compatibility with HPC environments allows for efficient, large-scale data processing at near real-time speeds, making it suitable for high-throughput applications in materials science. By successfully applying GRATEv2 to a substantial PCDTBT dataset, we demonstrated its efficacy in rapidly extracting critical structural features such as d-spacing, orientation, and shape metrics. This capability is particularly valuable for advancing research in organic electronics, where precise nanoscale characterization is essential for optimizing material properties.

Overall, GRATEv2 addresses key limitations of existing HRTEM analysis methods by providing a fast, adaptable, and user-friendly tool that enhances the efficiency and reliability of microstructural characterization. By making GRATEv2 open-source, we aim to facilitate its adoption and further development by the research community. Future work could involve extending GRATEv2 to other material systems and incorporating additional analytical capabilities, thereby broadening its applicability and impact in the field of materials characterization.

Methods

HRTEM Sample Preparation and Measurement Method

PCDTBT was synthesized using previously published procedures for Suzuki polycondensation in a Schlenk reactor flask[42, 43]. Sigma-Aldrich supplied all reactants. Polymerization occurred between 9-(9-heptadecanyl)-9H-carbazole-2,7-diboronic acid bis(pinacol) ester and 4,7-bis(2-bromo-5-thienyl)-2,1,3-benzothiadiazole in toluene, with equimolar amounts of each monomer. All other synthesis and purification procedures remained unaltered compared to the cited sources. The synthesis product was characterized by H1^{1} nuclear magnetic resonance at 500 MHz[44].

5 mg/mL solutions of PCDTBT and chlorobenzene (Sigma-Aldrich) were prepared in a nitrogen glovebox and mixed overnight on a hotplate at 45°C. Silicon wafers were sonicated in acetone for 20 minutes, then isopropanol for 20 minutes. The wafers then underwent 20 minutes of ultraviolet light ozonation. A PEDOT:PSS (Clevios P and H.C. Starck) and water solution were then spin-coated onto the clean substrates at 4000 RPM for 2 minutes. The samples were then brought into the glovebox, and the heated PCDTBT solution was spin-coated on top of the PEDOT:PSS layer at 800 RPM for 2 minutes. The sample was then cut into squares and floated off in the water, and samples were collected on copper TEM grids. Samples were left to dry overnight and then annealed inside a nitrogen glovebox.

High-resolution imaging experiments were conducted on the Titan Krios microscope at the Penn State Materials Characterization Laboratory. The accelerating voltage was 300 kV, and the detector was a Falcon 3EC direct electron detector in counted mode. Regions of interest were spaced 2.5 µm apart and visually inspected on the atlas image for tears and defects before acquisition. The spot size was set to 5, and autofocus was done at 300 kx magnification before being increased to 470 kx for acquiring a 2.5 second exposure. The microscope produced a 650 nm beam with a dose rate of 50 e/Ųs.

Quantities of Interest

The inputs to the algorithm are 1) the HRTEM image 2) Approx d-spacings of the crystals to detect 3) The resolution of the image 4) Process parameters. The algorithm outputs the segmentation result for each input and a CSV file containing the feature details as listed in Table 2 and Table 3.

The distance between the crystals are defined as below:

Dmetric=dcenter−centerr1+r2(1)D_{\mathrm{metric}} = \frac{d_{\mathrm{center-center}}}{r_1 + r_2} \tag*{(1)}
Ddirect=dcenter−center(2)D_{\mathrm{direct}} = d_{\mathrm{center-center}} \tag*{(2)}
SymbolDescription
DmetricD_{\mathrm{metric}}Metric distance: A dimensionless measure of the relative separation between two crystals, calculated by dividing the center-to-center distance by the sum of the radii of the crystals.
DdirectD_{\mathrm{direct}}Direct distance (nm): The actual center-to-center distance between two crystals, measured in nanometers (nm).
dcenter−centerd_{\mathrm{center-center}}Center-to-center crystal distance (nm): The straight-line distance between the centers of two crystals, measured in nanometers.
r1,r2r_1, r_2Radii of the crystals (nm): The radii of each crystal in nanometers, where r1r_1 represents the radius of the first crystal and r2r_2 represents the radius of the second.

Table 7 (PDF p. 14).

Image Processing Algorithm

Figure 9 illustrates the flowchart of GRATEv2 algorithm along with the associated process parameters. The process parameters are indicated in blue boxes, while the intermediate steps corresponding to Figure 10 are marked with brown numeric annotations.

Algorithm flowchart with parameters

Figure 9. Algorithm flowchart with parameters. Blue boxes are the process parameters and brown boxes are the intermediate steps shown in Figure 10.

Figure 10. Visualization of intermediate and final algorithm outputs.

The algorithm parameters are classified into two categories: primary parameters and secondary parameters. The primary parameters are independent values provided by the user based on the dataset’s imaging conditions and material properties, specifically the crystal d-spacing value and the image resolution. These parameters scale the key process parameters, enabling the algorithm to generalize across different imaging conditions and crystals of interest. They may vary significantly between datasets. Secondary parameters correspond to individual steps of the algorithm and are set to optimal values; they may require fine-tuning when changing datasets.

The algorithm is designed with several key objectives in mind: to generate a clear and concise representation of the essential information in the image; to enhance data quality by filtering out noise and irrelevant details; to convert the processed image data into a graph structure for advanced analysis; to use graph algorithms to identify clusters of polymer backbones forming crystallites; and to apply Fourier transform techniques to determine the d-spacing values.

Each step of the algorithm is detailed below, along with its associated parameters.

  1. Initially, blurring is applied to smooth the image and reduce sharpness, which helps in minimizing noise and preparing the image for subsequent processing. This operation is performed multiple times, with the number of blurring iterations specified as a parameter. Following blurring, histogram equalization is employed to increase the image contrast, making it sharper and enhancing the distinction between polymer chains and the background. Improved contrast facilitates more accurate thresholding in the subsequent step.

  1. The next step involves image thresholding using Otsu’s method to create a binary representation of the image, where polymer chains appear as black regions and the background as white. The result of this step is shown in Figure 10b. After thresholding, morphological closing and opening operations are performed to remove small black and white spots considered as noise. The kernel sizes for closing and opening are parameters that do not depend on the d-spacing and are set based on the noise characteristics of the image.

  1. Subsequently, we perform skeletonization and branching. Skeletonization reduces the polymer regions to single-pixel-wide lines, representing the skeleton of the polymer chains. Branching breaks the skeleton at junctions where three or more connections occur, resulting in branched skeletonized representations called backbones. The output of this step is depicted in Figure 10c.

  1. To refine the data, we proceed to segment and filter the backbones. Backbones shorter than a certain pixel length are considered noise and are filtered out; the threshold length is a parameter proportional to the d-spacing. The remaining backbones are segmented into uniform lengths, referred to as bones, for consistent evaluation. The division size is also a parameter proportional to the d-spacing. The result after filtering and division is shown in Figure 10d.

  1. For each bone, we perform ellipse construction by fitting an ellipse to its pixel locations using the scikit-image library [45]. This provides the major and minor axes of the bone, which are used for further analysis. Ellipses are preferred because they help filter out non-linear bones (curved structures not part of crystalline

regions) and facilitate the creation of a graph representation where each ellipse serves as a node.

  1. To identify the crystalline regions, we apply ellipse aspect ratio filtering. Crystalline regions are composed of linear polymer backbones, so we filter out curved bones by setting a threshold on the ellipse aspect ratio (major axis length divided by minor axis length). Bones with aspect ratios above this threshold are retained for further analysis. This step’s output is shown in Figure 10e.

  1. A graph is then constructed where each ellipse represents a node. An edge is created between two nodes if the distance between their centers is less than the adjacency distance parameter (proportional to the d-spacing) and the angle between their major axes is less than the adjacency angle parameter (also proportional to the d-spacing). The graph is stored as an adjacency matrix.

Using this graph representation, we perform clustering of adjacent bones by identifying connected components using a depth-first search algorithm. Clusters with a node count below a certain threshold are considered noise and discarded. The remaining clusters represent detected crystalline regions. For each cluster, a convex hull and an alpha shape are computed to segment the crystal region. Properties such as area, centroid, major and minor axis lengths, and orientation are evaluated. Additionally, a line indicating the direction of the crystal pattern is plotted. The output after clustering is shown in Figure 10f, and the final crystal detection is displayed in Figure 10g.

Finally, we perform d-spacing evaluation for each detected crystal region. The largest possible square region within the crystal is selected, and a Fast Fourier Transform (FFT) is performed to transform the spatial domain into the frequency domain. Band-pass filtering is applied to remove frequencies outside the range of interest. The location of the peak frequency above a set threshold is used to calculate the exact d-spacing value and the orientation of the crystal pattern. The frequency threshold is a parameter. The evaluation of d-spacing using FFT is illustrated in Figure 11. The algorithm outputs include visualizations such as convex hulls and alpha shapes of detected crystal regions overlaid on the original image, as well as quantitative data like area, centroid, major and minor axis lengths, orientation, and d-spacing values for each crystal. All features are saved in CSV files for further analysis. These outputs provide valuable insights into the material’s microstructure and can aid in understanding material properties more effectively.

Evaluation of d-spacing for the crystal using the Fast Fourier Transform

Figure 11. Evaluation of d-spacing for the crystal using the Fast Fourier Transform

The algorithm is implemented in Python, utilizing the scikit-image library for image processing and SciPy for computational functions. A configuration file is used to input all relevant parameters, dataset paths, and result paths. The code is tested on Ubuntu Linux-based local systems and HPC servers. It supports batch processing and multiprocessing, allowing for efficient analysis of large datasets. Results are stored in a well-organized directory structure, with options to save intermediate outputs for debugging purposes. Comprehensive documentation is provided, detailing code usage and parameter settings. A summary of the primary and secondary parameters used in the algorithm is provided in Appendix § 10. These parameters are optimized for detecting crystals in the HRTEM dataset of PCDTBT organic photovoltaic materials.

Bayesian Optimization

Optimizing hyperparameters in image processing algorithms is crucial for enhancing performance, especially in complex tasks like automated detection of crystalline domains in HRTEM images. Traditional methods of parameter tuning, such as grid search or manual adjustment, can be inefficient and may not guarantee optimal results due to the high dimensionality and computational expense. To address this challenge, we integrated Bayesian optimization into our framework, GRATEv2, to systematically and efficiently identify the optimal set of parameters.

Bayesian optimization is a sequential design strategy for global optimization of black-box functions that are expensive to evaluate especially due to large hyperparameter space [46]. It builds a probabilistic model of the objective function and uses it to select the most promising hyperparameters to evaluate next, balancing exploration and exploitation.

Mathematical Formulation

Let x∈X⊆Rd\mathbf{x} \in\mathcal{X} \subseteq\mathbb{R}^{d} denote a vector of dd hyperparameters, and let f:X→Rf : \mathcal{X} \to\mathbb{R} be the objective function that maps hyperparameters to a scalar performance metric. Our goal is to find the hyperparameters x∗\mathbf{x}^{*} that minimize the objective function:

x∗=arg⁡min⁡x∈X f(x).(3)\mathbf{x}^{*} = \underset{\mathbf{x}\in\mathcal{X}}{\arg\min}\, f(\mathbf{x}). \tag*{(3)}

In our case, f(x)f(\mathbf{x}) is the negative mean Intersection over Union (IoU) score between the detected crystalline regions and the ground truth annotations, as we aim to maximize the IoU.

Gaussian Process Surrogate Model

Bayesian optimization relies on a surrogate model to approximate the objective function. We use a Gaussian process (GP) prior [47] over functions to model f(x)f(\mathbf{x}). A GP is defined by its mean function m(x)m(\mathbf{x}) and covariance function k(x,x′)k(\mathbf{x}, \mathbf{x}'):

f(x)∼GP(m(x),k(x,x′)).(4)f(\mathbf{x}) \sim\mathcal{GP}\left(m(\mathbf{x}), k(\mathbf{x}, \mathbf{x}')\right). \tag*{(4)}

We assume a zero mean function m(x)=0m(\mathbf{x}) = 0 without loss of generality, and select a suitable covariance function, such as the squared exponential (radial basis function) kernel:

k(x,x′)=σf2exp⁡(−12(x−x′)TL−1(x−x′)),(5)k(\mathbf{x}, \mathbf{x}') = \sigma_f^2 \exp\left(-\frac{1}{2}(\mathbf{x} - \mathbf{x}')^{\mathrm{T}}\mathbf{L}^{-1}(\mathbf{x} - \mathbf{x}')\right), \tag*{(5)}

where σf2\sigma_f^2 is the signal variance and L\mathbf{L} is a diagonal matrix of length-scale parameters ℓi2\ell_i^2, controlling the smoothness of the function along each dimension.

Posterior Distribution

Given a set of nn observed data points Dn={(xi,yi)}i=1nD_n = \{(\mathbf{x}_i, y_i)\}_{i=1}^n, where yi=f(xi)+ϵiy_i = f(\mathbf{x}_i) + \epsilon_i and ϵi∼N(0,σn2)\epsilon_i \sim\mathcal{N}(0, \sigma_n^2) is Gaussian observation noise, the GP posterior predictive distribution at a new point x∗\mathbf{x}_* is given by:

μn(x∗)=k∗T(K+σn2I)−1y,(6)\mu_n(\mathbf{x}_*) = \mathbf{k}_*^{\mathrm{T}}(\mathbf{K} + \sigma_n^2\mathbf{I})^{-1}\mathbf{y}, \tag*{(6)}
σn2(x∗)=k(x∗,x∗)−k∗T(K+σn2I)−1k∗,(7)\sigma_n^2(\mathbf{x}_*) = k(\mathbf{x}_*, \mathbf{x}_*) - \mathbf{k}_*^{\mathrm{T}}(\mathbf{K} + \sigma_n^2\mathbf{I})^{-1}\mathbf{k}_*, \tag*{(7)}

where k∗=[k(x1,x∗),…,k(xn,x∗)]T\mathbf{k}_* = [k(\mathbf{x}_1, \mathbf{x}_*), \ldots, k(\mathbf{x}_n, \mathbf{x}_*)]^{\mathrm{T}}, K\mathbf{K} is the n×nn \times n covariance matrix with [K]ij=k(xi,xj)[\mathbf{K}]_{ij} = k(\mathbf{x}_i, \mathbf{x}_j), and y=[y1,…,yn]T\mathbf{y} = [y_1, \ldots, y_n]^{\mathrm{T}}.

Acquisition Function

An acquisition function α(x;Dn)\alpha(\mathbf{x}; D_n) guides the selection of the next evaluation point by quantifying the utility of sampling at x\mathbf{x}. We use the Expected Improvement (EI) acquisition function [48], defined as:

αEI(x)=E[max⁡(0,fmin⁡−f(x))],(8)\alpha_{\mathrm{EI}}(\mathbf{x}) = \mathbb{E}\left[\max(0, f_{\min} - f(\mathbf{x}))\right], \tag*{(8)}

where fmin⁡f_{\min} is the minimum observed value of the objective function. Under the GP posterior, the EI can be computed analytically:

αEI(x)=(fmin⁡−μn(x))Φ(fmin⁡−μn(x)σn(x))+σn(x)ϕ(fmin⁡−μn(x)σn(x)),(9)\alpha_{\mathrm{EI}}(\mathbf{x}) = (f_{\min} - \mu_n(\mathbf{x}))\Phi\left(\frac{f_{\min} - \mu_n(\mathbf{x})}{\sigma_n(\mathbf{x})}\right) + \sigma_n(\mathbf{x})\phi\left(\frac{f_{\min} - \mu_n(\mathbf{x})}{\sigma_n(\mathbf{x})}\right), \tag*{(9)}

where μn(x)\mu_n(\mathbf{x}) and σn(x)\sigma_n(\mathbf{x}) are the posterior mean and standard deviation from Eqs. (6) and (7), Φ(⋅)\Phi(\cdot) is the standard normal cumulative distribution function, and ϕ(⋅)\phi(\cdot) is the standard normal probability density function.

Optimization Loop

As shown in Figure 12, the Bayesian optimization algorithm proceeds iteratively through initialization, surrogate model updates, and acquisition function optimization.

Flowchart of the Bayesian Optimization Loop

Figure 12. Flowchart of the Bayesian Optimization Loop. The process iteratively updates the surrogate model and selects new hyperparameters to evaluate until convergence criteria are met.

The Bayesian optimization algorithm begins with the initialization step, where the objective function is evaluated at an initial set of hyperparameters {xi}i=1n0\{x_i\}_{i=1}^{n_0}, often chosen via a space-filling design like Latin hypercube sampling. Next, a Gaussian Process (GP) surrogate model is fitted to the observed data Dn\mathcal{D}_n, capturing our current understanding of the objective function.

Using this surrogate model, the algorithm maximizes the acquisition function to find the next promising set of hyperparameters:

xn+1=arg max⁡x∈Xα(x;Dn).(10)x_{n+1} = \underset{x \in\mathcal{X}}{\operatorname{arg\,max}} \alpha(x; \mathcal{D}_n). \tag*{(10)}

The objective function is then evaluated at xn+1x_{n+1} to obtain yn+1y_{n+1}, and the data is augmented:

Dn+1=Dn∪{(xn+1,yn+1)}.(11)\mathcal{D}_{n+1} = \mathcal{D}_n \cup\{(x_{n+1}, y_{n+1})\}. \tag*{(11)}

The algorithm checks if the convergence criterion is met (e.g., maximum iterations reached or negligible improvement observed). If not, the process loops back to updating the surrogate model with the new data, as depicted in Figure 12. This iterative process continues until convergence, resulting in the optimal set of hyperparameters.

Parameter Space

We optimized 13 hyperparameters of the image processing algorithm, each within specified bounds informed by prior knowledge as shown in Table 5. The ranges were chosen to cover a wide search space while ensuring that the parameters were physically meaningful and relevant to the task of detecting crystalline regions in HRTEM images.

HyperparameterRangeType
blur_iteration[5, 20]Integer
Blur_kernel_propCons[0.1, 0.5]Real
closing_k_size[1, 20]Integer
opening_k_size[1, 20]Integer
pixThresh_propCons[0.0, 1.0]Real
ellipse_len_propCons[0.5, 5.0]Real
ellipse_aspect_ratio[2.0, 7.0]Real
thresh_dist_propCons[1.0, 5.0]Real
thresh_theta[5.0, 15.0]Real
cluster_size[1, 10]Integer
dspace_bandpass[0.1, 0.5]Real
powSpec_peak_thresh[1.0, 1.5]Real
thresh_area_factor[1.0, 5.0]Real

Table 5. Hyperparameters and their ranges used in Bayesian optimization.

Objective Function Evaluation

Figure 13 shows example inputs for the Bayesian optimization training. We used a total of 13 annotated HRTEM images for training. The ground truth annotations are created using VGG annotator tool [49]. For each set of hyperparameters xx, the objective function f(x)f(x) was evaluated as follows:

Bayesian optimization training input, input images, ground truth annotations, and ground truth masks

Figure 13. Bayesian optimization training input i.e. input images, ground truth annotations, and ground truth masks. Each row represents a different sample from the dataset.

  1. Run the image processing algorithm with parameters xx on the annotated HRTEM images.

  1. Generate binary masks of the detected crystalline regions.

  1. Compute the IoU between the detected masks and the ground truth masks:

IoU⁡(x)=∣Detected∩Ground Truth∣∣Detected∪Ground Truth∣(12)\operatorname{IoU}(x) = \frac{\lvert\mathrm{Detected} \cap\mathrm{Ground\ Truth} \rvert}{\lvert\mathrm{Detected} \cup\mathrm{Ground\ Truth} \rvert} \tag*{(12)}

where ∣⋅∣\lvert\cdot\rvert denotes the cardinality (number of pixels in this context).

  1. Set the objective function value as f(x)=−IoU⁡(x)f(x) = -\operatorname{IoU}(x).

Algorithm Execution

Our implementation of the Gaussian Process-based Bayesian optimization algorithm for hyperparameter tuning is based on the scikit-optimize library [50]. The library provides a user-friendly interface through the gp_minimize function for Bayesian optimization and supports various acquisition functions, surrogate models, and optimization strategies. We used the gp_minimize function with the following key parameters:

  • Acquisition Function: Expected Improvement (EI).

  • Number of Calls: 200 total evaluations of the objective function.

  • Initial Points: 10 random initial evaluations to seed the GP model.

  • Random State: Seeded for reproducibility.

Evaluation of Data Sufficiency

Determining the optimal amount of data to collect is crucial for experimentalists. Collecting too little data can compromise the reliability of results, while collecting excessive data may not yield additional insights and can waste valuable resources. Establishing a reliable stopping criterion for data collection ensures that resources are utilized efficiently without sacrificing statistical significance.

An effective data sufficiency metric should possess certain desirable features:

  1. Sensitivity to Distribution Changes: It should accurately reflect changes in the underlying data distribution as more data is collected.

  1. Scale Interpretability: The metric should provide a quantitative measure that is interpretable and can be related to practical thresholds for decision-making.

  1. Applicability to Empirical Distributions: It should be suitable for comparing empirical distributions derived from finite samples.

  1. Metric Properties: The measure should satisfy the properties of a mathematical metric, such as non-negativity, identity of indiscernibles, symmetry, and triangle inequality, to ensure consistent and meaningful comparisons.

In this study, we introduce a stopping criterion based on the Wasserstein distance to assess data sufficiency. The Wasserstein distance, also known as the Earth Mover’s Distance, quantifies the difference between two probability distributions by measuring the minimum "cost" of transforming one distribution into the other. It is particularly well-suited for our purposes because it satisfies all the desired features of a data sufficiency metric listed above.

Mathematical Definition of the Wasserstein Distance Let PP and QQ be two probability distributions on the real line with cumulative distribution functions (CDFs) FP(x)F_P(x) and FQ(x)F_Q(x), respectively. The pp-th order Wasserstein distance Wp(P,Q)W_p(P,Q) between PP and QQ is defined as:

Wp(P,Q)=(∫−∞∞∣FP−1(u)−FQ−1(u)∣p du)1/p(13)W_p(P,Q) = \left(\int_{-\infty}^{\infty} \left|F_P^{-1}(u)-F_Q^{-1}(u)\right|^p\,du\right)^{1/p} \tag*{(13)}

where FP−1(u)F_P^{-1}(u) and FQ−1(u)F_Q^{-1}(u) are the quantile functions (inverse CDFs) of PP and QQ, and p≥1p \ge1. For p=1p=1, the first-order Wasserstein distance simplifies to:

W1(P,Q)=∫01∣FP−1(u)−FQ−1(u)∣ du(14)W_1(P,Q) = \int_0^1 \left|F_P^{-1}(u)-F_Q^{-1}(u)\right|\,du \tag*{(14)}

Alternatively, for discrete empirical distributions derived from finite samples, the first-order Wasserstein distance between two sets of observations {xi}i=1n\{x_i\}_{i=1}^{n} and {yj}j=1m\{y_j\}_{j=1}^{m} can be computed by sorting the observations and calculating the average absolute difference between the sorted values:

W1(Pn,Qm)=1N∑k=1N∣x(k)−y(k)∣(15)W_1(P_n,Q_m) = \frac{1}{N}\sum_{k=1}^{N}\left|x_{(k)}-y_{(k)}\right| \tag*{(15)}

where N=min⁡(n,m)N=\min(n,m), and x(k),y(k)x_{(k)}, y_{(k)} are the ordered statistics (sorted data).

The Wasserstein distance’s ability to capture differences between distributions, even when they have overlapping support, makes it ideal for assessing the convergence of empirical data distributions as more data is collected. By monitoring the Wasserstein distance between successive data samples, we can determine when the distributions have converged, indicating that the data collection process has reached a point of diminishing returns.

Data Availability

The dataset used in this work will be made available on request.

Code Availability

The software developed for this paper is available at https://bitbucket.org/baskargroup/gratev2.

Acknowledgements

Funding from the National Science Foundation under Award DMREF 2323716 and the Office of Naval Research under Awards N00014-19-1-2453, and N00014-23-1-2001 are gratefully acknowledged.

Conflict of Interest Statement

The authors have no competing interests to declare that are relevant to the content of this article.

Appendix

Optimum Parameters

The Bayesian optimization based optimum parameters and manually selected parameters of GRATEv2 for our dataset are given in Table 6.

ParameterDescriptionManually Selected ValueBayesian Optimized Value
dspace_nmd-Spacing in nm1.91.9
pix_2_nmpixels per nm78.578.5
blur_iterationBlurring Iterations1520
Blur_kernel_propConsKernel Size Blurring*0.150.12
closing_k_sizeKernel Size Closing152
opening_k_sizeKernel Size Opening172
pixThresh_propConsThreshold Pixel Length*0.630.74
ellipse_len_propConsUniform Breaking Length*1.504.03
ellipse_aspect_ratioThreshold Ellipse Aspect Ratio5.004.38
thresh_dist_propConsAdjacency Distance*2.001.36
thresh_thetaAdjacency Angle (degrees)10.0013.96
cluster_sizeThreshold Cluster Size79
dspace_bandpassBand Pass Filter Size0.200.44
powSpec_peak_threshPower Spectrum Threshold1.151.00
thresh_area_factorArea Threshold Factor4.002.79

Table 6. Comparison of Manually selected and Bayesian Optimized Parameters

    • Proportionality constant to dd − Spacing.

  • dspace_nm and pix_2_nm are user inputs and not optimized.

T-statistic for Data Sufficiency

To evaluate the statistical significance of the improvement, we performed a paired t-test on the IoU scores from the two sets of parameters. The null hypothesis (H0H_0) is that there is no difference in the mean IoU scores between the manual and Bayesian-optimized parameters. Let did_i be the difference in IoU scores for each image, defined as di=IoUBayesian,i−IoUManual,id_i = \mathrm{IoU}_{\mathrm{Bayesian},i} - \mathrm{IoU}_{\mathrm{Manual},i}.

dˉ=1n∑i=1ndi=0.1413(16)\bar{d} = \frac{1}{n} \sum_{i=1}^{n} d_i = 0.1413 \tag*{(16)}
sd=1n−1∑i=1n(di−dˉ)2=0.0489(17)s_d = \sqrt{\frac{1}{n-1} \sum_{i=1}^{n}(d_i-\bar{d})^2} = 0.0489 \tag*{(17)}

The t-statistic is computed as:

t=dˉsd/n=0.14130.0489/6=0.14130.01997=7.074.(18)t = \frac{\bar{d}}{s_d/\sqrt{n}} = \frac{0.1413}{0.0489/\sqrt{6}} = \frac{0.1413}{0.01997} = 7.074. \tag*{(18)}

With n−1=5n-1 = 5 degrees of freedom, the critical t-value at a significance level of α=0.05\alpha= 0.05 (two-tailed) is approximately 2.571. Since t=7.074>2.571t = 7.074 > 2.571, we reject the null hypothesis and conclude that the improvement in IoU scores using Bayesian optimization is statistically significant.

Additional Validation Results

Figure A.1. Additional comparison of ground truth, manually selected parameters, and Bayesian-optimized parameters across different images.

References

  1. [1]D.B. Williams and C.B. Carter. Transmission Electron Microscopy: A Textbook for Materials Science. Number v. 1 in Cambridge library collection. Springer, 2009. ISBN 9780387765006. URL https://books.google.com/books?id=dXdrG39VtUoC.
  2. [2]A.G. Cullis and P.A. Midgley. Microscopy of Semiconducting Materials 2003. CRC Press, 2018. ISBN 9781351083089. URL https://books.google.com/books?id=5MUIEQAAQBAJ.DOI
  3. [3]R F Ergerton. Electron energy-loss spectroscopy in the tem. Reports on Progress in Physics, 72(1):016502, dec 2008. doi: 10.1088/0034-4885/72/1/016502. URL https://dx.doi.org/10.1088/0034-4885/72/1/016502.
  4. [4]J.C.H. Spence. High-Resolution Electron Microscopy. OUP Oxford, 2013. ISBN 9780191508400. URL https://books.google.com/books?id=PitoAgAAQBAJ.
  5. [5]T. Isabell, J. Brink, M. Kawasaki, B. Armbruster, I. Ishikawa, E. Okunishi, H. Sawada, Y. Okura, K. Yamazaki, T. Ishikawa, and et al. Development of a 200kv atomic resolution analytical electron microscope. Microscopy Today, 17(3):8–11, 2009. doi: 10.1017/S155192950005045.
  6. [6]J. Kirkland. Advanced Computing in Electron Microscopy. SpringerLink: Springer e-Books. Springer US, 2010. ISBN 9781441965332. URL https://books.google.com/books?id=YsCLlyaiNv0C.
  7. [7]Christoph J Brabec, Andreas Distler, Xiaoyan Du, Hans-Joachim Elgelhaaf, Jens Hauch, Thomas Heumueller, and Ning Li. Material strategies to accelerate opv technology toward a gw technology. Advanced energy materials, 10(43):2001864, 2020.DOI
  8. [8]Paul A Midgley and Rafal E Dunin-Borkowski. Electron tomography and holography in materials science. Nature materials, 8(4):271–280, 2009.DOI
  9. [9]Pal Toth, Arpad B. Palotas, Eric G. Eddings, Ross T. Whitaker, and JoAnn S. Lighty. A novel framework for the quantitative analysis of high resolution transmission electron micrographs of soot i. improved measurement of interlayer spacing. Combustion and Flame, 160(5):909–919, 2013. ISSN 0010-2180. doi: https://doi.org/10.1016/j.combustflame.2013.01.002. URL https://www.sciencedirect.com/science/article/pii/S0010218013000035.
  10. [10]Yihan Zhu, Jim Ciston, Bin Zheng, Xiaohe Miao, Cory Czarnik, Yichang Pan, Rachid Sougrat, Zhiping Lai, Chia-En Hsiung, Kexin Yao, et al. Unravelling surface and interfacial structures of a metal–organic framework by transmission electron microscopy. Nature materials, 16(5):532–536, 2017.DOI
  11. [11]A Sharma, T Kyotani, and A Tomita. A new quantitative approach for microstructural analysis of coal char using hrtem images. Fuel, 78(10):1203–1212, 1999.
  12. [12]Pal Toth. Nanostructure quantification of turbostratic carbon by hrtem image analysis: State of the art, biases, sensitivity and best practices. Carbon, 178:688–707, 2021.DOI
  13. [13]Balaji Sesha Sarath Pokuri, Jacob Stimes, Kathryn O’Hara, Michael L. Chabinyc, and Baskar Ganapathysubramanian. GRATE: a Framework and Software for Graph Based Analysis of Transmission Electron Microscopy Images of Polymer Films. Computational Materials Science, 163:1–10, 2019. ISSN 0927-0256. doi: 10.1016/j.commatsci.2019.02.030. URL https://www.sciencedirect.com/science/article/pii/S0927025619301041.DOI
  14. [14]Utkarsh Pratiush, Austin Houston, Sergei V Kalinin, and Gerd Duscher. Realizing smart scanning transmission electron microscopy using high performance computing. Review of Scientific Instruments, 95(10), 2024.
  15. [15]Ayana Ghosh, Kevin Roccapiore, Matthew G Boebinger, Debangshu Mukherjee, Anees Al-Najjar, Marshall Mcdonnell, Sergei V Kalinin, and Maxim Ziatdinov. Integrating high-performance computing with electron microscopy for scientific insights, 2024.DOI
  16. [16]Utkarsh Pratiush, Kevin M Roccapiore, Yongtao Liu, Sergei V Kalinin, and Gerd Duscher. Realizing smart stem via machine learning on remote high performance computer, 2024.DOI
  17. [17]Yael Tsarfati, Karen C Bustillo, Benjamin H Savitzky, Adam Marks, Iain McCulloch, Colin Ophus, Andrew M Minor, and Alberto Salleo. Charge-induced structural rearrangements in organic mixed ionic electronic conductors: A cryogenic 4d-stem study, 2024.DOI
  18. [18]Yael Tsarfati, Karen C Bustillo, Benjamin H Savitzky, Colin Ophus, Iain McCulloch, Alberto Salleo, and Andrew M Minor. Structural study of hydrated organic mixed ionic electronic conductors using cryogenic 4d-stem, 2023.
  19. [19]Yutong Bi, Yuan Tian, Mingjie Xu, Evgeniy Boltynuk, Leonardo Velasco Estrada, Horst Hahn, Jian Han, David J Srolovitz, and Xiaoqing Pan. In situ 4d-stem study on grain boundary dynamics in polycrystals, 2023.DOI
  20. [20]Robert Winkler, Alexander Zintler, Oscar Recalde-Benitez, Tianshu Jiang, Déspina Nasiou, Esmaeil Adabifiroozjaei, Philipp Schreyer, Taewook Kim, Ezter Piros, Nico Kaiser, et al. Texture transfer in dielectric layers via nanocrystalline networks: Insights from in situ 4d-stem. Nano Letters, 24(10):2998–3004, 2024.DOI
  21. [21]Bruno César da Silva, Zahra Sadre Momtaz, Eva Monroy, Hanako Okuno, Jean-Luc Rouviere, David Cooper, and Martien Ilse Den Hertog. Assessment of active dopants and p–n junction abruptness using in situ biased 4d-stem. Nano Letters, 22(23):9544–9550, 2022.DOI
  22. [22]Hsin-Yun Chao, Kartik Venkatraman, Saman Moniri, Yongjun Jiang, Xuan Tang, Sheng Dai, Wenpei Gao, Jianwei Miao, and Miaofang Chi. In situ and emerging transmission electron microscopy for catalysis research. Chemical Reviews, 123(13):8347–8394, 2023.DOI
  23. [23]Yael Tsarfati, Karen C Bustillo, Luke Balhorn, Tyler J Quill, Jennifer Donohue, Steven E Zeltmann, Benjamin Savitzky, Colin Ophus, Christopher J Takacs, Iain McCulloch, et al. Microstructural study of organic mixed ionic-electronic conductor thin films using 4d-stem and hrtem. Microscopy and Microanalysis, 28(S1):350–351, 2022.DOI
  24. [24]Nabila Masud, Jaydeep Rade, Md Hasibul Hasan Hasib, Adarsh Krishnamurthy, and Anwesha Sarkar. Machine learning approaches for improving atomic force microscopy instrumentation and data analytics. Frontiers in Physics, 12:1347648, 2024.
  25. [25]Maxim Ziatdinov, Ondrej Dyck, Artem Maksov, Xufan Li, Xiahan Sang, Kai Xiao, Raymond R Unocic, Rama Vasudevan, Stephen Jesse, and Sergei V Kalinin. Deep learning of atomically resolved scanning transmission electron microscopy images: chemical identification and tracking local transformations. ACS nano, 11(12):12742–12752, 2017.DOI
  26. [26]Catherine K. Groschner, Christina Choi, and Mary C. Scott. Machine Learning Pipeline for Segmentation and Defect Information from High-Resolution Transmission Electron Microscope Data. Microscopy and Microanalysis, 27(3):549–556, May 2021. doi: 10.1017/S1431927621000386.DOI
  27. [27]Jaydeep Rade, Soumik Sarkar, Anwesha Sarkar, and Adarsh Krishnamurthy. 3d reconstruction of protein complex structures using synthesized multi-view afm images. arXiv preprint arXiv:2211.4662, 2022.
  28. [28]Seyed Majid Azimi, Dominik Britz, Michael Engstler, Mario Fritz, and Frank Mücklich. Advanced steel microstructural classification by deep learning methods. Scientific reports, 8(1):2128, 2018.DOI
  29. [29]Jean-Noël Rouzaud and Christian Clinard. Quantitative high-resolution transmission electron microscopy: a promising tool for carbon materials characterization. Fuel Processing Technology, 77:229–235, 2002.
  30. [30]Árpád B Palotás, Lenore C Rainey, Christian J Feldermann, Adel F Sarofim, and John B Vander Sande. Soot morphology: An application of image analysis in high-resolution transmission electron microscopy. Microscopy research and technique, 33(3):266–278, 1996.pubmed.ncbi.nlm.nih.gov/8652885
  31. [31]Kuen Yehliu, Randy L Vander Wal, and André L Boehman. Development of an hrtem image analysis method to quantify carbon nanostructure. Combustion and flame, 158(9):1837–1851, 2011.DOI
  32. [32]So Hyun Park, Sungmin Park, Seungjin Lee, Jiho Kim, Hyungju Ahn, Bumjoon J. Kim, Boknam Chae, and Hae Jung Son. Development of highly efficient large area organic photovoltaic module: Effects of nonfullerene acceptor. Nano Energy, 77:105147, 2020. ISSN 2211-2855. doi: https://doi.org/10.1016/j.nanoen.2020.105147. URL https://www.sciencedirect.com/science/article/pii/S2211285520307254.
  33. [33]Álvaro Rodríguez-Rodríguez, Esther Rebollar, Tiberio A Ezquerra, Marta Castillejo, Jose V Garcia-Ramos, and Mari-Cruz García-Gutiérrez. Patterning of conjugated polymers by laser: Synergy of nanostructure formation in the all-polymer heterojunction p3ht/pcdtbt. Langmuir, 34(1):115–125, 2018.DOI
  34. [34]Zach M Bailey, Eric T Hoke, Rodrigo Noriega, Javier Dacuña, George F Burkhard, Jonathan A Bartelt, Alberto Salleo, Michael F Toney, and Michael D McGehee. Morphology-dependent trap formation in high performance polymer bulk heterojunction solar cells. Advanced Energy Materials, 1(5):954–962, 2011.DOI
  35. [35]Xinhui Lu, Htay Hlaing, David S Germack, Jeff Peet, Won Ho Jo, Denis Andrienko, Kurt Kremer, and Benjamin M Ocko. Bilayer order in a polycarbazole-conjugated polymer. Nature communications, 3(1):795, 2012.DOI
  36. [36]Ryan A Fair, Dhruv Gamdha, Joshua T Del Mundo, Abigail M Fenton, Agatha O’Connell, Karen C Bustillo, Andrew M Minor, Baskar Ganapathysubramanian, and Enrique D Gomez. Automated tem reveals intercrystalline correlations of conjugated polymers. Submitted, in review, 2025.DOI
  37. [37]Stefan Van der Walt, Johannes L Schönberger, Juan Nunez-Iglesias, François Boulogne, Joshua D Warner, Neil Yager, Emmanuelle Gouillart, and Tony Yu. scikit-image: image processing in python. PeerJ, 2:e453, 2014.DOI
  38. [38]Gary Bradski, Adrian Kaehler, et al. Opencv. Dr. Dobb’s journal of software tools, 3(2), 2000.
  39. [39]Juan Nunez-Iglesias, Adam J Blanch, Oliver Looker, Matthew W Dixon, and Leann Tilley. A new python library to analyse skeleton images confirms malaria parasite remodelling of the red blood cell membrane skeleton. PeerJ, 6:e4312, 2018.
  40. [40]Pauli Virtanen, Ralf Gommers, Travis E Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, et al. Scipy 1.0: fundamental algorithms for scientific computing in python. Nature methods, 17(3):261–272, 2020.DOI
  41. [41]Charles R Harris, K Jarrod Millman, Stéfan J Van Der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J Smith, et al. Array programming with numpy. Nature, 585(7825):357–362, 2020.DOI
  42. [42]Renxuan Xie, Melissa P. Aplan, Nicholas J. Caggiano, Albree R. Weisen, Tang Su, Christian Müller, Mo Segad, Ralph H. Colby, and Enrique D. Gomez. Local Chain Alignment via Nematic Ordering Reduces Chain Entanglement in Conjugated Polymers. Macromolecules, 51(24):10271–10284, 2018. doi: 10.1021/acs.macromol.8b01840.DOI
  43. [43]Renxuan Xie, Youngmin Lee, Melissa P. Aplan, Nicholas J. Caggiano, Christian Müller, Ralph H. Colby, and Enrique D. Gomez. Glass Transition Temperature of Conjugated Polymers by Oscillatory Shear Rheometry. Macromolecules, 50(13):5146–5154, 2017. doi: 10.1021/acs.macromol.7b00712.DOI
  44. [44]Ryan A Fair, Renxuan Xie, Youngmin Lee, Ralph H Colby, and Enrique D Gomez. Molecular weight characterization of conjugated polymers through gel permeation chromatography and static light scattering. ACS Applied Polymer Materials, 3(9):4572–4578, 2021.DOI
  45. [45]Stefan Van der Walt, Johannes L Schönberger, Juan Nunez-Iglesias, François Boulogne, Joshua D Warner, Neil Yager, Emmanuelle Gouillart, and Tony Yu. Scikit-image: Image processing in Python. PeerJ, 2:e453, 2014.DOI
  46. [46]Peter I Frazier. A tutorial on bayesian optimization. arXiv preprint arXiv:1807.02811, 2018.arxiv.org/abs/1807.02811
  47. [47]Carl Edward Rasmussen. Gaussian processes in machine learning. In Summer school on machine learning, pages 63–71. Springer, 2003.DOI
  48. [48]Donald R Jones, Matthias Schonlau, and William J Welch. Efficient global optimization of expensive black-box functions. Journal of Global optimization, 13:455–492, 1998.
  49. [49]Abhishek Dutta and Andrew Zisserman. The VIA annotation software for images, audio and video. In Proceedings of the 27th ACM International Conference on Multimedia, MM ’19, New York, NY, USA, 2019. ACM. ISBN 978-1-4503-6889-6/19/10. doi: 10.1145/3343031.3350535. URL https://doi.org/10.1145/3343031.3350535.
  50. [50]T Head, M Kumar, H Nahrstaedt, G Louppe, and I Shcherbatyi. scikit-optimize/scikit-optimize (v0. 9.0). Zenodo. Available online: https://zenodo.org/record/5565057#. Y8Y-iRVBxPY (accessed on 1 July 2021), 2021.

Paper details

vlnc:W_bzkj4d97
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents