STEM image restoration using convolutional neural networks and its impact on the quantification of atomic structures.

Session

DHA.1 - Deep learning for analysis and interpretation of microscopy imaging data

Authors

Thomas Friedrich (1, 2), Ivan Lobato (1, 2), Sandra Vam Aert (1, 2)

Affiliations

1. EMAT
2. NANOlab Center of Excellence

Keywords

Convolutional Neural Networks

Distortions

Image Restoration

Quantitative electron microscopy

Scanning transmission electron microscopy

Abstract text

Summary
We trained a Deep Convolutional Neural Network (CNN) to correct image distortions in scanning transmission electron microscopy(STEM) data using a "blind restoration" approach. The effects of different distortion types were explored and quantified for different scenarios. The proposed approach shows impressive restoration results even for combinations of high levels of distortions for both, periodic and non-periodic specimens. We show that errors in the determination of atomic column positions, as well as scattering cross-sections, can be significantly reduced. We demonstrate that the variance for quantitative measures can be reduced even beyond the theoretical information limit, governed by the Cramér-Rao-lower bound, employing this restoration method.

Introduction
The image resolution of modern scanning transmission electron microscopes (STEM) has reached the sub-Angstrom range, enabling microscopists to directly image individual atoms or atomic columns. Beyond merely visual interpretations, atomic-resolution STEM images can be used to extract quantitative measures, such as atomic column positions, scattering cross-sections (SCS) and a number of derived quantities, such as the number of atoms in a nanoparticle or strain distributions [1]. Reliable determination of these measures is an important cornerstone for materials engineering at the nanoscale. Statistical parameter estimation theory was successfully employed to obtain accurate and precise estimates of atomic column positions [1] and SCS enabling one to locate atomic columns and to count atoms with single-atom sensitivity [3]. The reliability of these estimates depends on the quality of the underlying STEM image, which is naturally limited by distortions such as X- and Y-jitter and fast scan distortion and a mixture of counting- and thermal noise [2]. These effects are in practice particularly severe, when investigating beam sensitive materials and light elements and may make it very difficult to draw quantitative conclusions from STEM data [4].

This reasoning has been motivating the development of image post-processing and restoration approaches for both TEM and STEM data [6-8,11]. To correct these image distortions by means of classical image processing techniques, they need to be individually identified and corrected, resulting in a variety of specialised, complex and computationally expensive algorithms.
We show that deep convolutional neural networks can be used to denoise and correct image distortions in STEM data without any particular constraint, prior knowledge or additional user input within a few seconds even for 4k-by-4k pixel images using a blind restoration approach. Further, we determine the improvements in accuracy and precision for quantitative data analysis resulting from this image restoration and demonstrate how strongly individual distortions influence quantitative measurements in STEM experiments.

Methods
For the architecture of the model, we employed the concept of a fully convolutional deep neural network, with a stack of 12 convolutional layers with 64 filters and rectified-linear-units (Re-LUs) as element-wise non-linearities. The network was trained on a dataset of 20 million synthetic images using the Adam-optimizer initially, followed by an optimization employing the Stochastic-Gradient-Descent(SGD) method with a learning rate annealing scheme.

To investigate the effects of distortion correction on quantitative measurements of annular dark field (ADF)-STEM images, ground truth images were simulated using the multislice algorithm[9], then subjected to different modelled distortions and restored using the CNN as illustrated in figure 1. Scan line distortions (X- and Y-jitter) were modelled as scan line shifts in x- and y-direction, mathematically expressed as series of related terms[10], which introduce local random variations, as well as a long-range effect. Fast scan distortion arises from a finite detector response time. At short dwell times, the electron beam scans too fast for the detector, resulting in smearing along the scan line. The main contribution to noise in STEM images is the Poisson distributed counting noise. A combination of these effects changes an image as seen in figure 1a to the distorted version in figure 1b.

Atomic column positions and SCSs were retrieved using statistical parameter estimation theory [1] for both, the raw and restored images for 100 noise/distortion-setting realisations of each setup. The results were then evaluated by means of the variance of observations as illustrated in figure 2 or mean absolute error (MAE) and mean absolute percentage error (MAPE) as in figure 3.

a) Ground truth	b) Distorted	c) Restored

Figure 1: Example images of a Pt crystal in [110] zone axis orientation with a (111)-twin boundary. The image 1a was distorted by a mixture of Y-Jitter, X-Jitter, Fast scan distortion and Poisson noise at a dose of 5e3 Å^-2

Results/Conclusions
Figure 2 shows the variances of the column positions and SCSs measured on a FCC Pt crystal in [001] zone axis orientation over a thickness of 75 unit cells for two dose levels. The images were subjected only to Poisson noise in order to allow a comparison to the theoretical lower bound on the variance (Cramér-Rao-Bound)[1] which could indeed be overcome by the proposed method.

Figure 2: Variances and CRLB of atomic column position and scattering cross-section determinations.

For the example of a Pt crystal with twin boundary shown in figure 1, we determined the improvements in quantitative measures for different dose settings and a mixture of scan line distortions (X- and Y-Jitter), fast-scan distortion and noise, resembling a practically meaningful/common scenario. In figure 3 it can be seen that the mean error of atomic column position determinations can be reduced by ≈3pm, which is especially interesting for applications such as strain distribution measurements. The mean percentage error of the SCS is shown to be reduced by up to ≈3% which would translate directly into improvements in the accuracy with which atoms can be counted.

Figure 3: Quantitative results for a mixture of distortions and different doses for the example shown in figure 1.

In conclusion, our restoration approach provides a very valuable and easy to use tool to enhance the quality of STEM images not only for visualisation purposes but also as an important pre-processing step for various quantitative measurements as well as for advanced atomic-resolution 3D characterisation methods [2].

References

[1] A. De Backer, K. H. van den Bos, W. Van den Broek, J. Sijbers, S. Van Aert, StatSTEM: An efficient approach for accurate and precise model-based quantification of atomic resolution electron microscopy images, Ultramicroscopy 171 (2016) 104–116.

[2] T. Altantzis, I. Lobato, A. De Backer, A. B ́ech ́e, Y. Zhang, S. Basak, M. Porcu, Q. Xu, A. S ́anchez-Iglesias, L. M. Liz-Marz ́an, G. Van Tendeloo, S. Van Aert, S. Bals, Three-dimensional Quantification of the Facet Evolution of Pt Nanoparticles in a Variable Gaseous Environment, Nano Letters 19 (2019) 477–481

[3] S. Van Aert, A. De Backer, G. T. Martinez, B. Goris, S. Bals, G. Van Tendeloo, A. Rosenauer, procedure to count atoms with trustworthy single-atom sensitivity, Physical Review B - Condensed Matter and Materials Physics 87 (2013) 1–6.

[4]  J. Fatermans, A. J. Den Dekker, K. Müller-Caspary, I. Lobato, C. M. O’Leary, P. D. Nellist, S. Van Aert, Single Atom Detection from Low Contrast-to-Noise Ratio Electron Microscopy Images, Physical Review Letters 121 (2018) 56101.

[5] B. Bajic, J. Lindblad, N. Sladoje, Blind restoration of images degraded with mixed poisson-Gaussian noise with application in transmission electron microscopy, in: 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI), IEEE, 2016, pp. 123–127.

[6] N. Braidy, Y. Le Bouar, S. Lazar, C. Ricolleau, Correcting scanning instabilities from images of periodic structures, Ultramicroscopy 118 (2012) 67–76.

[7] N. Nakanishi, T. Yamazaki, A. Rečnik, M. Čeh, M. Kawasaki, K. Watanabe, M. Shiojiri, Retrieval process of high-resolution HAADF-STEM images, Journal of Electron Microscopy 51 (2002) 383–390.

[8] A. M. Sanchez, P. L. Galindo, S. Kret, M. Falke, R. Beanland, P. J. Goodhew, An approach to the systematic distortion correction in aberration-corrected HAADF images,
Journal of Microscopy 221 (2006) 1–7.

[9] I. Lobato, D. Van Dyck, MULTEM: A new multislice program to perform accurate and fast electron diffraction and imaging simulations using Graphics Processing Units with CUDA, Ultramicroscopy 156 (2015) 9–17.

 [10] G.  Walker,   On  Periodicity in  Series of  Related  Terms, Proceedings of the Royal Society of  London.  Series  A,  Containing Papers of a  Mathematical and PhysicalCharacter 131 (1931) 518–532.

 [11] L. Jones, P. D. Nellist, Identifying and correcting scan noise and drift in the scanning transmission electron microscope, Microscopy and Microanalysis,  Volume 19, Cambridge University Press, 2013, pp. 1050–1060.