Partial Scanning Transmission Electron Microscopy with Deep Learning

Jeffrey M Ede,Richard Beanland

doi:10.1038/s41598-020-65261-0

Abstract

Compressed sensing algorithms are used to decrease electron microscope scan time and electron beam exposure with minimal information loss. Following successful applications of deep learning to compressed sensing, we have developed a two-stage multiscale generative adversarial neural network to complete realistic 512 × 512 scanning transmission electron micrographs from spiral, jittered gridlike, and other partial scans. For spiral scans and mean squared error based pre-training, this enables electron beam coverage to be decreased by 17.9× with a 3.8% test set root mean squared intensity error, and by 87.0× with a 6.2% error. Our generator networks are trained on partial scans created from a new dataset of 16227 scanning transmission electron micrographs. High performance is achieved with adaptive learning rate clipping of loss spikes and an auxiliary trainer network. Our source code, new dataset, and pre-trained models are publicly available.

Highlights

Aberration corrected scanning transmission electron microscopy (STEM) can achieve imaging resolutions below 0.1 nm, and locate atom columns with pm precision[1,2]
Where pre-trained models are unavailable for transfer learning[24], artificial neural networks (ANNs) are typically trained, validated and tested with large, carefully partitioned machine learning datasets[25,26] so that they are robust to general use
Jittered gridlike scans would be difficult to produce with a conventional system, which would suffer variations in dose and distortions due to limited beam deflector response

Summary

Introduction

Aberration corrected scanning transmission electron microscopy (STEM) can achieve imaging resolutions below 0.1 nm, and locate atom columns with pm precision[1,2]. Where pre-trained models are unavailable for transfer learning[24], artificial neural networks (ANNs) are typically trained, validated and tested with large, carefully partitioned machine learning datasets[25,26] so that they are robust to general use This often requires at least a few thousand examples. To train an ANN to complete STEM images from partial scans, an ideal dataset might consist of a large number of pairs of partial scans and corresponding high-quality, low noise images, taken with an aberration-corrected STEM. Jittered gridlike scans would be difficult to produce with a conventional system, which would suffer variations in dose and distortions due to limited beam deflector response These idealized scan paths serve as useful inputs to demonstrate the capabilities of our approach. Meaningful distances can be hand-crafted or learned automatically by considering differences between features imagined by discriminators for real and generated images[34,35]

Discussion

Conclusion