Enabling deeper learning on big data for materials informatics applications

Dipendra Jha,Vishu Gupta,Alok Choudhary,Logan Ward,Ankit Agrawal,Christopher Wolverton,Zijiang Yang,Ian Foster,Wei-Keng Liao

doi:10.1038/s41598-021-83193-1

Abstract

The application of machine learning (ML) techniques in materials science has attracted significant attention in recent years, due to their impressive ability to efficiently extract data-driven linkages from various input materials representations to their output properties. While the application of traditional ML techniques has become quite ubiquitous, there have been limited applications of more advanced deep learning (DL) techniques, primarily because big materials datasets are relatively rare. Given the demonstrated potential and advantages of DL and the increasing availability of big materials datasets, it is attractive to go for deeper neural networks in a bid to boost model performance, but in reality, it leads to performance degradation due to the vanishing gradient problem. In this paper, we address the question of how to enable deeper learning for cases where big materials data is available. Here, we present a general deep learning framework based on Individual Residual learning (IRNet) composed of very deep neural networks that can work with any vector-based materials representation as input to build accurate property prediction models. We find that the proposed IRNet models can not only successfully alleviate the vanishing gradient problem and enable deeper learning, but also lead to significantly (up to 47%) better model accuracy as compared to plain deep neural networks and traditional ML techniques for a given input materials representation in the presence of big data.

Highlights

The crystal graph convolutional neural networks by incorporating information of the Voronoi tessellated crystal structure, explicit 3-body correlations of neighboring constituent atoms, and an optimized chemical representation of interatomic bonds in the crystal graph, for accelerated materials discovery
Dataset from Open Quantum Materials Database (OQMD) is composed of 341,443 unique compositions, with their density functional theory (DFT)-computed materials properties comprising of formation enthalpy, band gap, energy per atom, and volume, as of May 2018
We demonstrated that the presented DL model architectures leveraging the proposed approach are versatile in their vector-based model input by evaluating prediction models for different materials properties using different combination of vector-based material representations: composition-derived 145 physical attributes and/or 126 structure-derived attributes with(out) 86 raw elemental fractions

Summary

Introduction

The crystal graph convolutional neural networks by incorporating information of the Voronoi tessellated crystal structure, explicit 3-body correlations of neighboring constituent atoms, and an optimized chemical representation of interatomic bonds in the crystal graph, for accelerated materials discovery. There have been several approaches to deal with the performance degradation issue due to vanishing and/or exploding gradient problem To address this issue for deep neural networks with fully connected layers, we present a novel approach of residual learning based on He et al.[38]; other approaches[39,52] will result in a tremendous increase in the number of model parameters, which could lead to GPU memory issues. We find that the use of individual residual learning in IRNet models can successfully alleviate the vanishing gradient problem and enable deeper learning, and leads to significantly (up to 47%) better model accuracy as compared to traditional ML techniques for a given input materials representation, when big data is available. IRNet leverages a simple and intuitive approach of individual residual learning to build the deep neural networks without using any domain-dependent model engineering, which makes it attractive for the materials scientists, and for other domain scientists in general to leverage it for their predictive modeling tasks on available big datasets

Methods

Results

Conclusion