Elsevier

Ultramicroscopy

Volume 242, December 2022, 113625
Ultramicroscopy

SIM-STEM Lab: Incorporating Compressed Sensing Theory for Fast STEM Simulation

https://doi.org/10.1016/j.ultramic.2022.113625Get rights and content

Highlights

  • Compressive sensing can reduce the run-time of STEM simulation.

  • Faster simulation can lead a way to real-time image interpretation.

  • Image quality is functionally identical with significant increases in speed.

  • Allows low-end machines to perform simulations much faster through inpainting.

Abstract

Recently it has been shown that precise dose control and an increase in the overall acquisition speed of atomic resolution scanning transmission electron microscope (STEM) images can be achieved by acquiring only a small fraction of the pixels in the image experimentally and then reconstructing the full image using an inpainting algorithm. In this paper, we apply the same inpainting approach (a form of compressed sensing) to simulated, sub-sampled atomic resolution STEM images. We find that it is possible to significantly sub-sample the area that is simulated, the number of g-vectors contributing the image, and the number of frozen phonon configurations contributing to the final image while still producing an acceptable fit to a fully sampled simulation. Here we discuss the parameters that we use and how the resulting simulations can be quantifiably compared to the full simulations. As with any Compressed Sensing methodology, care must be taken to ensure that isolated events are not excluded from the process, but the observed increase in simulation speed provides significant opportunities for real time simulations, image classification and analytics to be performed as a supplement to experiments on a microscope to be developed in the future.

Introduction

The simulation of atomic resolution scanning transmission electron microscopy (STEM) images [1], [2], [3], [4] has advanced significantly in recent years, to the point that the methodology is now capable of identifying single atom changes in structure and composition from a direct comparison to experimental images [5], [6], [7], [8]. Simulations are now used to help understand the structure-property relationships in a wide range of beam tolerant materials, interfaces, grain boundaries and defects [9], [10], [11], [12], [13], [14], [15], [16], [17], [18]. These simulations, however, typically involve computationally expensive formalisations where the core underlying approach is to include as many possible contributions/configurations as possible to ensure the best match to experimental images [[19], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39],40,41]. In addition to the significant computational power required and run time necessary to make sure the simulations have converged to best match the experiment; this approach runs into severe limitations when the comparison experimental image quality is poor [[42], [43], [44], [45], [46]] . This means that if a material, interface, or defect is susceptible to electron beam damage, it is hard to develop precise simulations of these structures.

For these damage limited acquisitions, there are currently two approaches to improving the inherent low signal-to-noise ratios that are being applied that make use of artificial intelligence: one is by using compressive sensing/inpainting [[47], [48], [49]] and the other is by classifying and learning features from a large number of low-quality images [[50], [51], [52]]. Could something similar be applied to simulations? Rather than performing one perfect simulation, could we run a set of smaller less precise (sub-sampled) simulations and use machine learning/artificial intelligence to classify and quantify the structure for comparison to experimental results? Even further, if we use machine learning to classify images, is there really a difference between the use of a simulated image and an experimental image to solve the structure at hand?

In this paper we will focus only on one use of artificial intelligence for image/simulation reconstruction, sub-sampling and inpainting, i.e., compressive sensing (CS), which has been shown to reduce the time and electron dose rate needed to form STEM images [[42], [43], [44]]. Compressive sensing STEM involves forming images that are directly subsampled at the time of acquisition (Fig. 1). An inpainting algorithm is then used to fill in gaps in the sub-sampled data, with missing information inferred from the subsampled data through, e.g., a combination of a dictionary learning or a sparsity pursuit algorithm (Fig. 2). To estimate a full image from simulated subsampled image, we use an unsupervised dictionary learning method. Compared to its supervised counterparts, which require a suitable pre-defined dictionary. Dictionary learning algorithms produce a dictionary of basic signal patterns, which is learned from the data, through a sparse linear combination with a set of corresponding weights. This dictionary is then used in conjunction with a sparse pursuit algorithm to inpaint the pixels of each overlapping patch which when combined form a full image [53]. In this paper, the Beta-Process Factor Analysis via Expectation Maximisation (BPFA-EM) method will be used to inpaint the subsampled data, and the reader is referred to [54] for more detail. We do not claim that BPFA is the state-of-the-art blind inpainting algorithm, however it is one of the top performing models outside of deep-learning methods. BPFA does not require prior knowledge of measurement noise, unlike its counterpart, i.e., K-SVD [40]. This is beneficial for real CS-STEM where noise level estimation is not always known, but for these simulations, noise is not considered prior to inpainting.

To determine whether it is possible to use the same inpainting approaches to the generation of image simulations, we first must examine what is involved in simulating atomic resolution images. In this regard, the multislice approach [41,55] is a common method for simulating the projected electron wavefunction, where the three-dimensional atomic potential is divided into a series of two-dimensional slices. The weak phase object approximation (WPOA) is assumed where only the phase of the incident electron wave is slightly modified [34], but not the amplitude. To account for electron-phonon interactions, the frozen phonon model is often used [33], [34], [35]. The intensity average of a series of multislice calculations is taken for different possible configurations of atomic coordinates (Fig. 5), and the number of frozen phonon configurations is linear with the run-time. In STEM simulations, this series of calculations must be solved at each probe position, which ultimately defines the resolution of the resulting simulation. Therefore, the run-time of STEM simulation scales not only with resolution, but the theoretical accuracy of the desired calculation, with a 128 × 128 pixel simulation taking on the order of 1 × 104 seconds with only 10 frozen phonon configurations, and a sample thickness of 1nm. Typically, the user must compromise between the run-time, accuracy, and size of the simulated image or must invest in high-end machines which can speed up the process through GPU parallelization [20], [21], [22], [23], [24], [25], such as that used by the MULTEM software referenced and used in this paper. It is in overcoming this compromise between run-time and accuracy, that inpainting can provide the most benefit.

In this paper, we will discuss how combining inpainting with STEM simulations, it is possible to form a new workflow for structural determination which can significantly reduce the run-time required for functionally identical results. This development allows real-time simulations to be performed, which is the first step in the rapid interpretation, classification, and analysis of images and potentially the future development of artificial intelligent (AI) STEM, AISTEM.

Section snippets

Methods

In the following section, three possible methods to reduce the run-time of STEM simulations will be presented and explained. The first is based on probe subsampling, the second is related to the number of frozen phonon configurations (FPC) used, and the third is based on the optimisation of the maximum reciprocal space vector that contributes to the simulated image. In all cases, the simulations were performed using MULTEM through MATLAB on a remote server equipped with an Intel Xeon Gold 6128

Results

The methods above were tested using the ZK-5 zeolite sample shown in Fig. 2. The patch size used for all sampling ratios is [2×2], and the image size is 128 × 128 pixels (8Å x 8Å). The metrics used in each of the methods are the structural similarity (SSIM) [56], and peak signal-to-noise ratio (PSNR) [57]. The performance of the methods is summarised in Fig. 7, Fig. 8, Fig. 9. The reference simulation used 32 frozen phonon configurations, and a maximum reciprocal space vector of 10.92Å1, and

Conclusion

In conclusion, we have shown that it is possible to significantly reduce the run-time of STEM simulation by leveraging the theory of compressed sensing. Furthermore, the results of using these three methods in combination yield results that are functionally similar, often identical to the fully simulated images. It is important to note that these methods are not expected to yield greater accuracy but are to be used in conjunction with experimental results, which are themselves inherently

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

This work was performed in the Albert Crewe Centre (ACC) for Electron Microscopy, a shared research facility (SRF) fully supported by the University of Liverpool. This work was also funded by the EPSRC Centre for Doctoral Training in Distributed Algorithms (EP/S023445/1) and Sivananthan Labs. The authors would also like to recognise the efforts of Ivan Lobato et al. for their development of MULTEM, without which this research would have not been possible. The work of Dr. Mounib Bahri

References (57)

  • J Fatermans

    Single atom detection from low contrast-to-noise ratio electron microscopy images

    Phys. Rev. E

    (2018)
  • A de Backer

    Hybrid statistics-simulations based method for atom-counting from ADF STEM images

    Ultramicroscopy

    (2017)
  • DA Muller

    Structure and bonding at the atomic scale by scanning transmission electron microscopy

    Nat. Mater.

    (2009)
  • P Abellan et al.

    Factors influencing quantitative liquid (scanning) transmission electron microscopy

    Chem. Commun.

    (2014)
  • Z Wang

    In situ STEM-EELS observation of nanoscale interfacial phenomena in all-solid-state batteries

    Nano Lett.

    (2016)
  • MJ Zachman et al.

    Cryo-STEM mapping of solid–liquid interfaces and dendrites in lithium-metal batteries

    Nature

    (2018)
  • C Sun et al.

    Analysis of crystal defects by scanning transmission electron microscopy (STEM) in a modern scanning electron microscope

    Adv. Struct. Chem. Imaging

    (2019)
  • C Ophus

    Efficient linear phase contrast in scanning transmission electron microscopy with matched illumination and detector interferometry

    Nat. Commun.

    (2016)
  • PM Pelz et al.

    Low-dose cryo electron ptychography via non-convex Bayesian optimization

    Sci. Rep.

    (2017)
  • K Fukao et al.

    Scanning transmission electron microscopy simulation for multi-domain barium titanate

    J. Phys. Soc. Jpn.

    (Mar. 2021)
  • PJ Phillips et al.

    Diffraction contrast STEM of dislocations: imaging and simulations

    Ultramicroscopy

    (Aug. 2011)
  • K Ishizuka

    A practical approach for STEM image simulation based on the FFT multislice method

    Ultramicroscopy

    (Feb. 2002)
  • K Ishizuka et al.

    A new theoretical and practical approach to the multislice method

    Acta Crystallogr., Sect. A

    (Sep. 1977)
  • EJ Kirkland

    Advanced Computing in Electron Microscopy, Second

    (2010)
  • EJ Kirkland et al.

    Simulation of annular dark field stem images using a modified multislice method

    Ultramicroscopy

    (Jan. 1987)
  • PA Stadelmann

    EMS - a software package for electron diffraction analysis and HREM image simulation in materials science

    Ultramicroscopy

    (Jan. 1987)
  • P Stadelmann

    Image analysis and simulation software in transmission electron microscopy

    Microsc. Microanal.

    (Sep. 2003)
  • R Kilaas, “MacTempas a program for simulating high resolution TEM images and diffraction patterns.,”...
  • Cited by (8)

    • Electron beam damages in zeolites: A review

      2024, Microporous and Mesoporous Materials
    View all citing articles on Scopus
    View full text