Deep learning detection of nanoparticles and multiple object tracking of their dynamic evolution during in situ ETEM studies

Khuram Faraz,Thomas Grenier,Christophe Ducottet,Thierry Epicier

doi:10.1038/s41598-022-06308-2

Abstract

In situ transmission electron microscopy (TEM) studies of dynamic events produce large quantities of data especially under the form of images. In the important case of heterogeneous catalysis, environmental TEM (ETEM) under gas and temperature allows to follow a large population of supported nanoparticles (NPs) evolving under reactive conditions. Interpreting properly large image sequences gives precious information on the catalytic properties of the active phase by identifying causes for its deactivation. To perform a quantitative, objective and robust treatment, we propose an automatic procedure to track nanoparticles observed in Scanning ETEM (STEM in ETEM). Our approach involves deep learning and computer vision developments in multiple object tracking. At first, a registration step corrects the image displacements and misalignment inherent to the in situ acquisition. Then, a deep learning approach detects the nanoparticles on all frames of video sequences. Finally, an iterative tracking algorithm reconstructs their trajectories. This treatment allows to deduce quantitative and statistical features about their evolution or motion, such as a Brownian behavior and merging or crossing events. We treat the case of in situ calcination of palladium (oxide) / delta-alumina, where the present approach allows a discussion of operating processes such as Ostwald ripening or NP aggregative coalescence.

Highlights

IntroductionIn the case of a high density, heterogeneous motion, or when specific events, such as disappearance, merging or splitting of particles, can occur, the data association problem becomes very challenging
We adapt the Continuous Energy Minimization (CEM) tracker proposed by Milan et al.[56] to account for the specific apparent behavior of imaged nanoparticles
Recorded video-type sequences of STEM micrographs in environmental TEM (ETEM) allow to track the nanoparticle trajectories during their dynamic evolution on a heterogeneous support with contrast variations, subject to a proper registration of the image series as it was performed with the affine alignment procedure

Summary

Introduction

In the case of a high density, heterogeneous motion, or when specific events, such as disappearance, merging or splitting of particles, can occur, the data association problem becomes very challenging This situation has received particular attention in the field of live-cell imaging where time-lapse sequences are a nalyzed[47]. The work of Jaqaman et al.[55] is among the few addressing complex situations with high particle density, particle motion heterogeneity, temporary particle disappearance, and particle merging and splitting Their algorithm first links particles between consecutive frames and links the resulting track segments into complete trajectories by solving a global combinatorial optimization problem. The original algorithm, being devoted to tracking humans, does not take into account the peculiar constraints we have with NPs observed in ETEM (mass conservation, disappearance, merging and splitting, etc.) and we propose some important improvements of this method

Objectives

Methods

Results

Discussion

Conclusion