SIM-STEM Lab: Incorporating Compressed Sensing Theory for Fast STEM Simulation
Introduction
The simulation of atomic resolution scanning transmission electron microscopy (STEM) images [1], [2], [3], [4] has advanced significantly in recent years, to the point that the methodology is now capable of identifying single atom changes in structure and composition from a direct comparison to experimental images [5], [6], [7], [8]. Simulations are now used to help understand the structure-property relationships in a wide range of beam tolerant materials, interfaces, grain boundaries and defects [9], [10], [11], [12], [13], [14], [15], [16], [17], [18]. These simulations, however, typically involve computationally expensive formalisations where the core underlying approach is to include as many possible contributions/configurations as possible to ensure the best match to experimental images [[19], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39],40,41]. In addition to the significant computational power required and run time necessary to make sure the simulations have converged to best match the experiment; this approach runs into severe limitations when the comparison experimental image quality is poor [[42], [43], [44], [45], [46]] . This means that if a material, interface, or defect is susceptible to electron beam damage, it is hard to develop precise simulations of these structures.
For these damage limited acquisitions, there are currently two approaches to improving the inherent low signal-to-noise ratios that are being applied that make use of artificial intelligence: one is by using compressive sensing/inpainting [[47], [48], [49]] and the other is by classifying and learning features from a large number of low-quality images [[50], [51], [52]]. Could something similar be applied to simulations? Rather than performing one perfect simulation, could we run a set of smaller less precise (sub-sampled) simulations and use machine learning/artificial intelligence to classify and quantify the structure for comparison to experimental results? Even further, if we use machine learning to classify images, is there really a difference between the use of a simulated image and an experimental image to solve the structure at hand?
In this paper we will focus only on one use of artificial intelligence for image/simulation reconstruction, sub-sampling and inpainting, i.e., compressive sensing (CS), which has been shown to reduce the time and electron dose rate needed to form STEM images [[42], [43], [44]]. Compressive sensing STEM involves forming images that are directly subsampled at the time of acquisition (Fig. 1). An inpainting algorithm is then used to fill in gaps in the sub-sampled data, with missing information inferred from the subsampled data through, e.g., a combination of a dictionary learning or a sparsity pursuit algorithm (Fig. 2). To estimate a full image from simulated subsampled image, we use an unsupervised dictionary learning method. Compared to its supervised counterparts, which require a suitable pre-defined dictionary. Dictionary learning algorithms produce a dictionary of basic signal patterns, which is learned from the data, through a sparse linear combination with a set of corresponding weights. This dictionary is then used in conjunction with a sparse pursuit algorithm to inpaint the pixels of each overlapping patch which when combined form a full image [53]. In this paper, the Beta-Process Factor Analysis via Expectation Maximisation (BPFA-EM) method will be used to inpaint the subsampled data, and the reader is referred to [54] for more detail. We do not claim that BPFA is the state-of-the-art blind inpainting algorithm, however it is one of the top performing models outside of deep-learning methods. BPFA does not require prior knowledge of measurement noise, unlike its counterpart, i.e., K-SVD [40]. This is beneficial for real CS-STEM where noise level estimation is not always known, but for these simulations, noise is not considered prior to inpainting.
To determine whether it is possible to use the same inpainting approaches to the generation of image simulations, we first must examine what is involved in simulating atomic resolution images. In this regard, the multislice approach [41,55] is a common method for simulating the projected electron wavefunction, where the three-dimensional atomic potential is divided into a series of two-dimensional slices. The weak phase object approximation (WPOA) is assumed where only the phase of the incident electron wave is slightly modified [34], but not the amplitude. To account for electron-phonon interactions, the frozen phonon model is often used [33], [34], [35]. The intensity average of a series of multislice calculations is taken for different possible configurations of atomic coordinates (Fig. 5), and the number of frozen phonon configurations is linear with the run-time. In STEM simulations, this series of calculations must be solved at each probe position, which ultimately defines the resolution of the resulting simulation. Therefore, the run-time of STEM simulation scales not only with resolution, but the theoretical accuracy of the desired calculation, with a 128 × 128 pixel simulation taking on the order of 1 × 104 seconds with only 10 frozen phonon configurations, and a sample thickness of 1nm. Typically, the user must compromise between the run-time, accuracy, and size of the simulated image or must invest in high-end machines which can speed up the process through GPU parallelization [20], [21], [22], [23], [24], [25], such as that used by the MULTEM software referenced and used in this paper. It is in overcoming this compromise between run-time and accuracy, that inpainting can provide the most benefit.
In this paper, we will discuss how combining inpainting with STEM simulations, it is possible to form a new workflow for structural determination which can significantly reduce the run-time required for functionally identical results. This development allows real-time simulations to be performed, which is the first step in the rapid interpretation, classification, and analysis of images and potentially the future development of artificial intelligent (AI) STEM, AISTEM.
Section snippets
Methods
In the following section, three possible methods to reduce the run-time of STEM simulations will be presented and explained. The first is based on probe subsampling, the second is related to the number of frozen phonon configurations (FPC) used, and the third is based on the optimisation of the maximum reciprocal space vector that contributes to the simulated image. In all cases, the simulations were performed using MULTEM through MATLAB on a remote server equipped with an Intel Xeon Gold 6128
Results
The methods above were tested using the ZK-5 zeolite sample shown in Fig. 2. The patch size used for all sampling ratios is , and the image size is 128 × 128 pixels (8Å x 8Å). The metrics used in each of the methods are the structural similarity (SSIM) [56], and peak signal-to-noise ratio (PSNR) [57]. The performance of the methods is summarised in Fig. 7, Fig. 8, Fig. 9. The reference simulation used 32 frozen phonon configurations, and a maximum reciprocal space vector of , and
Conclusion
In conclusion, we have shown that it is possible to significantly reduce the run-time of STEM simulation by leveraging the theory of compressed sensing. Furthermore, the results of using these three methods in combination yield results that are functionally similar, often identical to the fully simulated images. It is important to note that these methods are not expected to yield greater accuracy but are to be used in conjunction with experimental results, which are themselves inherently
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgements
This work was performed in the Albert Crewe Centre (ACC) for Electron Microscopy, a shared research facility (SRF) fully supported by the University of Liverpool. This work was also funded by the EPSRC Centre for Doctoral Training in Distributed Algorithms (EP/S023445/1) and Sivananthan Labs. The authors would also like to recognise the efforts of Ivan Lobato et al. for their development of MULTEM, without which this research would have not been possible. The work of Dr. Mounib Bahri
References (57)
Seeing the atoms more clearly: STEM imaging from the Crewe era to today
Ultramicroscopy
(2012)An introduction to the STEM
J. Ultrastruct. Res.
(1984)- et al.
High resolution imaging properties of the STEM
Ultramicroscopy
(1975) - et al.
High dose efficiency atomic resolution imaging via electron ptychography
Ultramicroscopy
(2019) - et al.
On the quantitativeness of EDS STEM
Ultramicroscopy
(2015) Dr. Probe: a software for high-resolution STEM image simulation
Ultramicroscopy
(2018)- et al.
MULTEM: a new multislice program to perform accurate and fast electron diffraction and imaging simulations using graphics processing units with CUDA
Ultramicroscopy
(2015) - et al.
Progress and new advances in simulating electron microscopy datasets using MULTEM
Ultramicroscopy
(2016) Scanning transmission electron microscopy
J. Microsc.
(1974)- et al.
Quantitative atomic resolution scanning transmission electron microscopy
Phys. Rev. E
(2008)
Single atom detection from low contrast-to-noise ratio electron microscopy images
Phys. Rev. E
Hybrid statistics-simulations based method for atom-counting from ADF STEM images
Ultramicroscopy
Structure and bonding at the atomic scale by scanning transmission electron microscopy
Nat. Mater.
Factors influencing quantitative liquid (scanning) transmission electron microscopy
Chem. Commun.
In situ STEM-EELS observation of nanoscale interfacial phenomena in all-solid-state batteries
Nano Lett.
Cryo-STEM mapping of solid–liquid interfaces and dendrites in lithium-metal batteries
Nature
Analysis of crystal defects by scanning transmission electron microscopy (STEM) in a modern scanning electron microscope
Adv. Struct. Chem. Imaging
Efficient linear phase contrast in scanning transmission electron microscopy with matched illumination and detector interferometry
Nat. Commun.
Low-dose cryo electron ptychography via non-convex Bayesian optimization
Sci. Rep.
Scanning transmission electron microscopy simulation for multi-domain barium titanate
J. Phys. Soc. Jpn.
Diffraction contrast STEM of dislocations: imaging and simulations
Ultramicroscopy
A practical approach for STEM image simulation based on the FFT multislice method
Ultramicroscopy
A new theoretical and practical approach to the multislice method
Acta Crystallogr., Sect. A
Advanced Computing in Electron Microscopy, Second
Simulation of annular dark field stem images using a modified multislice method
Ultramicroscopy
EMS - a software package for electron diffraction analysis and HREM image simulation in materials science
Ultramicroscopy
Image analysis and simulation software in transmission electron microscopy
Microsc. Microanal.
Cited by (8)
Electron beam damages in zeolites: A review
2024, Microporous and Mesoporous MaterialsTowards real-time STEM simulations through targeted subsampling strategies
2023, Journal of Microscopy