Training datasets

Recording atomic-resolution ADF-STEM datasets for a large library of crystal structures with known ground-truth labels is an extremely time-consuming project. Even though the high-resolution images with satisfying quality can be obtained regardless of the cost, it is not a trivial task to define the atomic column labels with high precision and accuracy. To mitigate this problem, we create a forward model that can simulate the experimental-like ADF-STEM images of different atomic structures from different crystallographic orientations with realistic noise models. In this way, the ground truth atomic positions are pre-defined. It is also time efficient to create an experimental-like ADF-STEM image set that comprises of a large number of spatial symmetries, atomic arrangements, zone axes, different noise levels and random backgrounds which can greatly improve the robustness of our models.

In this study, we used a simple linear imaging model which simulates ADF-STEM images by convolving the projected atomic potential of a material with the point spread function (PSF) of a scanning transmission electron microscope. Here, we only use the simplified version of the linear imaging model which disregards the three-dimensional shape of the point spread function because other than reducing contrast, it is a very subtle effect on atomic resolution images

and

here, we opt out using full quantum mechanical methods, such as Bloch-wave and multislice simulations, to simulate images because the simple linear imaging model we employ here is computationally much more affordable. (For STEM simulation, calculating an N×N-pixel image requires N×N’s multislice simulations. However, the computational complexity of our method is equal one single multislice simulation of a very thin sample.) Using the simple linear imaging model, we can render ten thousand 256-by-256 images within minutes whereas even with GPU acceleration it would still take days for the multislice simulation to compute them. For creating a static library, multislice simulation has its merit as it captures most of the scattering physics. However, for on-the-fly local training, the simple linear imaging model is more desirable because of its speed.

In addition, it has been shown that the apparent atomic column positions in ADF-STEM images may not always correspond to the actual atomic positions²⁷. This type of quantum phenomena is heavily crystal structure, thickness and orientation dependent. In addition, quantum mechanical simulation offers quantitatively correct column contrast in the simulated images which is one subtlety that can be compensated by other adjustments of the training sets. (The column contrast can be adjusted in our training images by changing the PSF and the background level.) Because our models only aim at reporting the apparent positions of the atomic columns, a simple linear imaging model is sufficient.

From the view of generating static libraries for the community, we have created two versions of the same library with one simulated by the simple linear imaging model²⁸ and one with the multislice simulation²⁹.

The second part of the forward model is the simulation of realistic noises in the ADF-STEM images. The primary sources of noise of ADF-STEM are the shot/Poisson noise (also known as counting noise) and the scan noise, which we will describe in detail as follows.

For a given pixel, the expected number of incoming electrons is calculated by n=tdwell×I/e. The counted electrons in this pixel follows the Poisson distribution, Pn=e-nnk/n! (Because photomultiplier has extremely high quantum efficiency, we ignore the propagation of additional noises.)

Random or periodic electromagnetic field or circuit level interreference can cause the beam to deviate away from the expected scanning position; therefore, the effect of the scan noise is a geometrical transformation of the ideal images. We denote the deviation vector by δi,j=(δxi,j,δyi,j) where i is the row number, and j is the column number. We define the horizontal direction is the fast scanning direction and the vertical direction is the slow scanning direction. Here, for simplicity, we assume that the beam deviation vector does not change when it scans through a horizontal row, i.e. the deviation vector δi,j=δi and δxi and δyj both follow the same normal distribution modulated by the periodic line frequency, i.e.

So, the final image is a transformation of the ideal image I₀ by

Some example images of how the noise model affect the images are shown in Fig. 1.

Synthetic images. (a) Images created by the simple linear imaging forward model with (b) synthetic shot and (c) scan noises.

To construct a training library with a variety of spatial symmetries, column contrast, and thickness effects, we have included images of the bulk structures of the following materials and orientations: Pt [001], Pt [110], NiO [001], NiO [110], SrTiO₃ [001] and [110], DyScO₃ [110], Si [110], graphene, amorphous graphene, single-layer MoS₂, rutile TiO₂ [001], [100] and [110], (Li)CoO₂ [010]. We have also included images of the faceted Pt nanocrystal to increase the robustness of finding atoms at boundaries and edges of nanoparticles and interfaces.

To enable robust and scale-free training we have included the following randomized operations in Table ​Table11 in the simulation of the training images and some example images are shown in Fig. 2.

Augmentation operations.

A few examples of the synthetic ADF-STEM images from the training library.

We have trained our model to perform atom segmentation, atomic-column Gaussian mapping, intensity-preserving super-resolution (deblur) processing, denoising and background removal. Their respective ground truth labels are shown in Fig. 3 and Table ​Table2.2. The width of the circular mask is defined by the full width at half maximum of the point spread function and the width of the Gaussian mask is 0.2 angstrom.

The (a) synthetic image and ground-truth labels for (b) intensity-preserving super-resolution (deblur) processing (c) atomic-column Gaussian mapping, (d) denoising, (e) denoising + background removal, (f) atomic-column segmentation.

Description of ground truth labels.

Radius of the circularMask label is defined as

r=0.5λαmax2+dsource22