Neural Network for Nanoscience Scanning Electron Microscope Image Recognition

Mohammad Hadi Modarres,Rossella Aversa,Regina Ciancio,Angelo Leto,Stefano Cozzini,Giuseppe Piero Brandino

doi:10.1038/s41598-017-13565-z

Abstract

In this paper we applied transfer learning techniques for image recognition, automatic categorization, and labeling of nanoscience images obtained by scanning electron microscope (SEM). Roughly 20,000 SEM images were manually classified into 10 categories to form a labeled training set, which can be used as a reference set for future applications of deep learning enhanced algorithms in the nanoscience domain. The categories chosen spanned the range of 0-Dimensional (0D) objects such as particles, 1D nanowires and fibres, 2D films and coated surfaces, and 3D patterned surfaces such as pillars. The training set was used to retrain on the SEM dataset and to compare many convolutional neural network models (Inception-v3, Inception-v4, ResNet). We obtained compatible results by performing a feature extraction of the different models on the same dataset. We performed additional analysis of the classifier on a second test set to further investigate the results both on particular cases and from a statistical point of view. Our algorithm was able to successfully classify around 90% of a test dataset consisting of SEM images, while reduced accuracy was found in the case of images at the boundary between two categories or containing elements of multiple categories. In these cases, the image classification did not identify a predominant category with a high score. We used the statistical outcomes from testing to deploy a semi-automatic workflow able to classify and label images generated by the SEM. Finally, a separate training was performed to determine the volume fraction of coherently aligned nanowires in SEM images. The results were compared with what was obtained using the Local Gradient Orientation method. This example demonstrates the versatility and the potential of transfer learning to address specific tasks of interest in nanoscience applications.

Highlights

The ability to identify and recognise specific features within images is of particular interest to scientists working with microscopy techniques
We mainly focused on the feature extraction technique, by retraining the final layers of an Inception-v312 deep convolutional neural network, implemented with the TensorFlow (TF) library[13], to perform the classification task on scanning electron microscope (SEM) images
We will present the scientific results achieved through the transfer learning approach we propose. These results address the initial part of the ambitious work we are doing within the NFFA-Europe project to provide the nanoscience community with a searchable database of SEM images

Summary

Introduction

The ability to identify and recognise specific features within images is of particular interest to scientists working with microscopy techniques. Neural networks have been employed for machine learning in a number of recent studies, extracting features from different types of microscope images. Recognition of cellular organisms from scanning probe microscopy images was shown using artificial neural networks[4]. Image recognition techniques can be a very powerful tool in nanoscience, where a large number of images are the typical outcome of characterization techniques such as scanning electron microscopy. A clear set of principles guiding this process has been published: data should be Findable, Accessible, Interoperable and Reusable (FAIR Guiding Principles[8]). Such an approach favors the idea that sharable data will validate findings, will promote their reuse, and will stimulate collaborations

Methods

Results

Conclusion