Potential Of Deep Neural Networks Research Articles

Automatic Knee Osteoarthritis Severity Grading using Deep Neural Networks: Comparative Analysis of Network Architectures and Optimization Functions

Knee osteoarthritis (OA) is a prevalent degenerative joint disease that requires accurate assessment of its severity for effective treatment planning. In this study, we propose an automatic knee OA severity-grading system based on deep neural networks. Specifically, we explore various network architectures, including VGG-16, VGG-19, ResNet-101, EfficientNet-B7, and EfficientNet-B6, along with different optimization functions such as SGD, ADAM, Nadam, AdamW, and AdaDelta. Furthermore, we investigate two loss functions, namely, the novel ordinal loss and the cross-entropy loss. The proposed system is evaluated on a carefully curated dataset, and comprehensive experimental settings are employed to ensure reliable results. Our findings indicate that the combination of the EfficientNet-B7 network with the Nadam optimizer yields the best performance, achieving an accuracy of 70.1% in knee OA severity grading. These results demonstrate the potential of deep neural networks in automating the grading process, offering a valuable tool for clinicians and researchers in the field of knee osteoarthritis management.

Deep learning for sleep analysis on children with sleep-disordered breathing: Automatic detection of mouth breathing events

IntroductionSleep-disordered breathing (SDB) can range from habitual snoring to severe obstructive sleep apnea (OSA). A common characteristic of SDB in children is mouth breathing, yet it is commonly overlooked and inconsistently diagnosed. The primary aim of this study is to construct a deep learning algorithm in order to automatically detect mouth breathing events in children from polysomnography (PSG) recordings.MethodsThe PSG of 20 subjects aged 10–13 years were used, 15 of which had reported snoring or presented high snoring and/or high OSA values by scoring conducted by a sleep technologist, including mouth breathing events. The separately measured mouth and nasal pressure signals from the PSG were fed through convolutional neural networks to identify mouth breathing events.ResultsThe finalized model presented 93.5% accuracy, 97.8% precision, 89% true positive rate, and 2% false positive rate when applied to the validation data that was set aside from the training data. The model's performance decreased when applied to a second validation data set, indicating a need for a larger training set.ConclusionThe results show the potential of deep neural networks in the analysis and classification of biological signals, and illustrates the usefulness of machine learning in sleep analysis.

Multistage linguistic conditioning of convolutional layers for speech emotion recognition

IntroductionThe effective fusion of text and audio information for categorical and dimensional speech emotion recognition (SER) remains an open issue, especially given the vast potential of deep neural networks (DNNs) to provide a tighter integration of the two.MethodsIn this contribution, we investigate the effectiveness of deep fusion of text and audio features for categorical and dimensional SER. We propose a novel, multistage fusion method where the two information streams are integrated in several layers of a DNN, and contrast it with a single-stage one where the streams are merged in a single point. Both methods depend on extracting summary linguistic embeddings from a pre-trained BERT model, and conditioning one or more intermediate representations of a convolutional model operating on log-Mel spectrograms.ResultsExperiments on the MSP-Podcast and IEMOCAP datasets demonstrate that the two fusion methods clearly outperform a shallow (late) fusion baseline and their unimodal constituents, both in terms of quantitative performance and qualitative behavior.DiscussionOverall, our multistage fusion shows better quantitative performance, surpassing alternatives on most of our evaluations. This illustrates the potential of multistage fusion in better assimilating text and audio information.

Unravelling the dissolution dynamics of silicate minerals by deep learning molecular dynamics simulation: A case of dicalcium silicate

Quantitative analyses of the thermodynamics and kinetics of silicate minerals dissolution at atomic level are difficult currently, both experimentally and computationally. Here, we apply the deep learning, enhancing sampling molecular dynamics and density functional theory to build up a deep neural network potential with quantum mechanics accuracy, which can determine the free energy surfaces, minimum free-energy reaction pathways and kinetic rates for Ca dissolution from the water/dicalcium silicate interface at different temperatures. We find that the Ca dissolution is a spontaneous reaction and follows different minimum free-energy reaction pathways at different temperatures. The dissolution time of the five-coordinated Ca ion is on the order of hundreds of seconds at ambient temperature and increases to the order of nanoseconds after heating. The relatively slow dissolution kinetics comparing to the low free energy barriers of each elementary reaction is attributed to the multi-directional and multi-step nature of the dissolution reaction. This new atomistic insight promotes a better understanding of the dicalcium silicate and cement hydration.

Development and Validation of Versatile Deep Atomistic Potentials for Metal Oxides.

Machine learning interatomic potentials powered by neural networks have been shown to readily model a gradient of compositions in metallic systems. However, their application to date on ionic systems tends to focus on specific compositions and oxidation states owing to their more heterogeneous chemical nature. Herein we show that a deep neural network potential (DNP) can model various properties of metal oxides with different oxidation states without additional charge information. We created and validated DNPs for AgxOy, CuxOy MgxOy, PtxOy, and ZnxOy, whereby each system was trained without any limitations on oxidation states. We illustrate how the database can be augmented to enhance the DNP transferability for a new polymorph, surface energies, and thermal expansion. In addition, we show that these potentials can correctly interpolate significant pressure and temperature ranges, exhibit stability over long molecular dynamics simulation time scales, and replicate nonharmonic thermal expansion, consistent with experimental results.

Accessing the thermal conductivities of Sb2Te3 and Bi2Te3/Sb2Te3 superlattices by molecular dynamics simulations with a deep neural network potential.

Phonon thermal transport is a key feature for the operation of thermoelectric materials, but it is challenging to accurately calculate the thermal conductivity of materials with strong anharmonicity or large cells. In this work, a deep neural network potential (NNP) is developed using a dataset based on density functional theory (DFT) and applied to describe the lattice dynamics of Sb2Te3 and Bi2Te3/Sb2Te3 superlattices. The lattice thermal conductivities of Sb2Te3 are first predicted using equilibrium molecular dynamics (EMD) simulations combined with an NNP and the results match well with experimental values. Then, through further exploration of weighted phase spaces and the Grüneisen parameter, we find that there is a stronger anharmonicity in the out-of-plane direction in Sb2Te3, which is the reason why the thermal conductivities are overestimated more in the out-of-plane direction than in the in-plane direction by solving the phonon Boltzmann transport equation (BTE) with only three-phonon scattering processes being considered. More importantly, the lattice thermal conductivities of Bi2Te3/Sb2Te3 superlattices with different periods are accurately predicted using non-equilibrium molecular dynamics (NEMD) simulations together with an NNP, which serves as a good example to explore the thermal transport physics of superlattices using a deep neural network potential.

Tuning the lattice thermal conductivity of Sb2Te3 by Cr doping: a deep potential molecular dynamics study.

Element doping is a prominent method for reducing the lattice thermal conductivity and optimizing the thermoelectric performance of materials in the thermoelectric field. However, determination of the thermal conductivity of element-doped systems is a challenging task, especially when the elements are randomly doped. In this work, a first-principles based deep neural network potential (NNP) is developed to investigate the lattice thermal transport properties of Cr-doped Sb2Te3 using molecular dynamics simulations. Compared with pure Sb2Te3, the thermal conductivity of orderly Cr-doped Sb2Te3 with Cr atoms locating at specific atomic layer positions decreases slightly in the in-plane direction, but sharply in the out-of-plane direction. The decrease of the low frequency phonon density of states and the enhancement of phonon scattering near 2.5 THz are the primary reasons for the decrease in the thermal conductivity of Cr-doped Sb2Te3, while the decrease of phonon velocity due to band flattening is the reason for the sharp decrease of thermal conductivity in the out-of-plane direction. Moreover, the thermal conductivities of randomly Cr-doped Sb2Te3 with different Cr concentrations are also investigated using the NNP. It is found that the thermal conductivities in both the in-plane and out-of-plane directions are reduced by 76% and 80%, respectively, for Sb36Cr36Te108. Furthermore, the influence of different Cr dopant arrays on the thermal conductivity of Sb2Te3 is also predicted using the NNP. Our work provides a good example for predicting the thermal conductivity of element-doped systems using the NNP combined with molecular dynamics simulations.

Temperature forecasting by deep learning methods

Abstract. Numerical weather prediction (NWP) models solve a system of partial differential equations based on physical laws to forecast the future state of the atmosphere. These models are deployed operationally, but they are computationally very expensive. Recently, the potential of deep neural networks to generate bespoke weather forecasts has been explored in a couple of scientific studies inspired by the success of video frame prediction models in computer vision. In this study, a simple recurrent neural network with convolutional filters, called ConvLSTM, and an advanced generative network, the Stochastic Adversarial Video Prediction (SAVP) model, are applied to create hourly forecasts of the 2 m temperature for the next 12 h over Europe. We make use of 13 years of data from the ERA5 reanalysis, of which 11 years are utilized for training and 1 year each is used for validating and testing. We choose the 2 m temperature, total cloud cover, and the 850 hPa temperature as predictors and show that both models attain predictive skill by outperforming persistence forecasts. SAVP is superior to ConvLSTM in terms of several evaluation metrics, confirming previous results from computer vision that larger, more complex networks are better suited to learn complex features and to generate better predictions. The 12 h forecasts of SAVP attain a mean squared error (MSE) of about 2.3 K2, an anomaly correlation coefficient (ACC) larger than 0.85, a structural similarity index (SSIM) of around 0.72, and a gradient ratio (rG) of about 0.82. The ConvLSTM yields a higher MSE (3.6 K2), a smaller ACC (0.80) and SSIM (0.65), and a slightly larger rG (0.84). The superior performance of SAVP in terms of MSE, ACC, and SSIM can be largely attributed to the generator. A sensitivity study shows that a larger weight of the generative adversarial network (GAN) component in the SAVP loss leads to even better preservation of spatial variability at the cost of a somewhat increased MSE (2.5 K2). Including the 850 hPa temperature as an additional predictor enhances the forecast quality, and the model also benefits from a larger spatial domain. By contrast, adding the total cloud cover as predictor or reducing the amount of training data to 8 years has only small effects. Although the temperature forecasts obtained in this way are still less powerful than contemporary NWP models, this study demonstrates that sophisticated deep neural networks may achieve considerable forecast quality beyond the nowcasting range in a purely data-driven way.

Using Neural Networks to Model Complex Mathematical Functions

Accurately modeling highly complex and irregular mathematical functions like fractals, chaos, and turbulence poses longstanding challenges. Traditional physics-based approaches often fail due to analytic intractability and extreme sensitivity. In this work, we pioneer the usage of long short-term memory (LSTM) recurrent neural networks for learning representations of such complex mathematical functions. We train custom-designed deep LSTM architectures on functions including the Lorenz attractor, Mandelbrot set, and Mackey-Glass delay differential equation. The networks achieve excellent quantitative performance across critical evaluation metrics like mean squared error and R-squared. Qualitative visualizations also demonstrate highly precise function replication and generalization. Comparisons to polynomial regression and multilayer perceptron baselines confirm the superiority of the LSTM modeling approach. Our results provide broader evidence for the potential of deep neural networks to emulate intricate mathematical functions, with transformative implications for overcoming modeling limitations across the sciences and engineering. We conclude by reflecting on current methodological limitations and identifying key open questions to guide future work at the intersection of mathematical modeling and machine learning.

ResGraphNet: GraphSAGE with embedded residual module for prediction of global monthly mean temperature

Data-driven prediction of time series is significant in many scientific research fields such as global climate change and weather forecast. For global monthly mean temperature series, considering the strong potential of deep neural network for extracting data features, this paper proposes a data-driven model, ResGraphNet, which improves the prediction accuracy of time series by an embedded residual module in GraphSAGE layers. The experimental results of a global mean temperature dataset, HadCRUT5, show that compared with 11 traditional prediction technologies, the proposed ResGraphNet obtains the best accuracy. The error indicator predicted by the proposed ResGraphNet is smaller than that of the other 11 prediction models. Furthermore, the performance on seven temperature datasets shows the excellent generalization of the ResGraphNet. Finally, based on our proposed ResGraphNet, the predicted 2022 annual anomaly of global temperature is 0.74722 °C, which provides confidence for limiting warming to 1.5 °C above pre-industrial levels.

Theoretical Investigation of the Electronic Spectra of Cadmium Chalcogenide 2D Nanoplatelets.

Although colloidal cadmium chalcogenide 2D nanoplatelets (NPLs) have recently demonstrated strongly enhanced one- and two-photon absorption (OPA, TPA) spectra, an understanding of the effects of quantum confinement in the lateral and vertical (thickness) dimensions is mostly lacking. In this work, we investigate theoretically CdS and CdSe NPLs passivated with formate and acetate ligands with thicknesses of two and three monolayers (MLs) and different lateral dimensions. Initial structures for CdS nanoplatelets were obtained using our recently developed deep neural network potential, and the low-energy geometries were subsequently optimized using density functional theory (DFT). Linear- and nonlinear-response calculations using time-dependent DFT (TDDFT) and the simplified Tamm-Dancoff approximation (sTDA) demonstrated good agreement between measured spectra and calculated TDDFT and sTDA spectra for 2 and 3 ML NPLs. The OPA red-shifts from 2 to 3 ML NPLs can be attributed to the electron delocalization in the lateral and vertical directions. TPA responses for CdS and CdSe NPLs were found to be dominated by weakly absorbing and forbidden OPA states.

Modeling Chemical Reactions in Alkali Carbonate-Hydroxide Electrolytes with Deep Learning Potentials.

We developed a deep potential machine learning model for simulations of chemical reactions in molten alkali carbonate-hydroxide electrolyte containing dissolved CO2, using an active learning procedure. We tested the deep neural network (DNN) potential and training procedure against reaction kinetics, chemical composition, and diffusion coefficients obtained from density functional theory (DFT) molecular dynamics calculations. The DNN potential was found to match DFT results for the structural, transport, and short-time chemical reactions in the melt. Using the DNN potential, we extended the time scales of observation to 2 ns in systems containing thousands of atoms, while preserving quantum chemical accuracy. This allowed us to reach chemical equilibrium with respect to several chemical species in the melt. The approach can be generalized for a broad spectrum of chemically reactive systems.

Accurate interatomic potential for the nucleation in liquid Ti-Al binary alloy developed by deep neural network learning method

For the traditional empirical potentials, the accuracy of chemical bonding in a wide temperature range is regrettable, and thereby they are not an appropriate choice to explore the nucleation process of crystals. As a developed superalloy in aerospace, the liquid structure and nucleation mechanism of Ti-Al alloy desire to be investigated to comprehend the solidification process deeply. This work develops an accurate deep neural network (DNN) potential for Ti-Al binary system by employing a machine-learning method. Then, the local structure and the nucleation process of liquid Ti-48 %Al and Ti-52 %Al (atomic percentage) alloys from the normal to the undercooled state are performed by molecular dynamics simulation with the DNN potential. It is found that the formation of the locally ordered structures depends on the Ti-Al atom pair during the solidification. The Honeycutt-Andersen indices of 1431,1541 and 1551 are dominant, and the liquid structure is mainly characterized by perfect and distorted icosahedral clusters. By Voronoi Polyhedron (VP) analysis, Ti-centered VPs favor the formation of body-centered-cubic (BCC) structure, while Al-centered VPs have the main contribution to the icosahedral structure. Besides, liquid Ti-52 %Al alloy prefers to form face-centered-cube (FCC) nuclei than BCC nuclei in a high undercooling, which is closely related to the ability to capture Ti-Al bonds. The favor of the nucleation of the γ phase causes the nuclei with non-equilibrium composition, comprised of Ti-rich BCC and Al-rich FCC coordinated atoms, respectively. Novelty, the FCC nuclei in liquid Ti-Al alloys are independently generated without the precursor of BCC coordinated atoms. High undercooling can transform the nucleation mechanism from two steps to one step.

Modeling the Solvation and Acidity of Carboxylic Acids Using an Ab Initio Deep Neural Network Potential.

Formic and acetic acid constitute the simplest of carboxylic acids, yet they exhibit fascinating chemistry in the condensed phase such as proton transfer and dimerization. The go-to method of choice for modeling these rare events have been accurate but expensive ab initio molecular dynamics simulations. In this study, we present a deep neural network potential trained using accurate ab initio data that can be used in tandem with enhanced-sampling methods to perform an efficient exploration of the free-energy surface of aqueous solutions of weak carboxylic acids. In particular, we show that our model captures proton dissociation and provides a good estimate of the pKa, as well as the dimerization of formic and acetic acid. This provides a suitable starting point for applications in different research areas where computational efficiency coupled with the accuracy of ab initio methods is required.

A tungsten deep neural-network potential for simulating mechanical property degradation under fusion service environment

Tungsten is a promising candidate material in fusion energy facilities. Molecular dynamics (MD) simulations reveal the atomistic scale mechanisms, so they are crucial for the understanding of the macroscopic property deterioration of tungsten under harsh and complex service environments. The interatomic potential used in the MD simulations is required to accurately describe a wide spectrum of relevant defect properties, which is by far challenging to the existing interatomic potentials. In this paper, we propose a new three-body embedding descriptor and hybridize it into the deep-potential (DP) framework, an end-to-end deep learning interatomic potential model. The potential model for tungsten, named DP-HYB, is trained with a database constructed by the concurrent learning method. The DP-HYB model is able to accurately predict elastic constants, stacking fault energy, the formation energies of free surfaces, and point defects, which are considered in the training dataset. It also accurately evaluates the formation energies of grain boundaries and prismatic loops, the core structure of screw dislocation, the Peierls barrier, and the transition path of the screw dislocation migration, which do not explicitly present in the training dataset. The DP-HYB is a good candidate for the atomistic simulations of tungsten property deterioration, especially those involving the mechanical property degradation under the harsh fusion service environment.

Analysis of Current Deep Learning Networks for Semantic Segmentation of Anatomical Structures in Laparoscopic Surgery.

Semantic segmentation of anatomical structures in laparoscopic videos is a crucial task to enable the development of new computer-assisted systems that can assist surgeons during surgery. However, this is a difficult task due to artifacts and similar visual characteristics of anatomical structures on the laparoscopic videos. Recently, deep learning algorithms have been showed promising results on the segmentation of laparoscopic instruments. However, due to the lack of large public datasets for semantic segmentation of anatomical structures, there are only a few studies on this task. In this work, we evaluate the performance of five networks, namely U-Net, U-Net++, DynUNet, UNETR and DeepLabV3+, for segmentation of laparoscopic cholecystectomy images from the recently released CholecSeg8k dataset. To the best of our knowledge, this is the first benchmark performed on this dataset. Training was performed with dice loss. The networks were evaluated on segmentation of 8 anatomical structures and instruments, performance was quantified through the dice coefficient, intersection over union, recall, and precision. Apart from the U-Net, all networks obtained scores similar to each other, with the U-Net++ being the network with the best overall score with a mean Dice value of 0.62. Overall, the results show that there is still room for improvement in the segmentation of anatomical structures from laparoscopic videos. Clinical Relevance- The results of this study show the potential of deep neural networks for the segmentation of anatomical structures in laparoscopic images which can later be incorporated into computer-aided systems for surgeons.

Comparative Analysis of Current Deep Learning Networks for Breast Lesion Segmentation in Ultrasound Images.

Automatic lesion segmentation in breast ultrasound (BUS) images aids in the diagnosis of breast cancer, the most common type of cancer in women. Accurate lesion segmentation in ultrasound images is a challenging task due to speckle noise, artifacts, shadows, and lesion variability in size and shape. Recently, convolutional neural networks have demonstrated impressive results in medical image segmentation tasks. However, the lack of public benchmarks and a standardized evaluation method hampers the networks' performance comparison. This work presents a benchmark of seven state-of-the-art methods for the automatic breast lesion segmentation task. The methods were evaluated on a multi-center BUS dataset composed of three public datasets. Specifically, the U-Net, Dynamic U-Net, Semantic Segmentation Deep Residual Network with Variational Autoencoder (SegResNetVAE), U-Net Transformers, Residual Feedback Network, Multiscale Dual Attention-Based Network, and Global Guidance Network (GG-Net) architectures were evaluated. The training was performed with a combination of the cross-entropy and Dice loss functions and the overall performance of the networks was assessed using the Dice coefficient, Jaccard index, accuracy, recall, specificity, and precision. Despite all networks having obtained Dice scores superior to 75%, the GG-Net and SegResNetVAE architectures outperform the remaining methods, achieving 82.56% and 81.90%, respectively. Clinical Relevance- The results of this study allowed to prove the potential of deep neural networks to be used in clinical practice for breast lesion segmentation also suggesting the best model choices.

Identifying Chirality in Line Drawings of Molecules Using Imbalanced Dataset Sampler for a Multilabel Classification Task.

Chirality, the ability of some molecules to exist as two non-superimposable mirror images, profoundly influences both chemistry and biology. Advances in deep learning enable the automatic recognition of chemical structure diagrams, however, studies on discovering the molecule chirality are scarce and the machine-readable molecular representations are not always sufficient to fully support the encoding of this important property. Here, we pretrained networks on a ChEMBL+ dataset (79641 molecules) and fine-tuned them for the binary classification of chirality (achiral/chiral) or multilabel chirality type classifications (none/centre/axial/planar). To address the label combination imbalanced problem in the multilabel task, the study proposed a Formulated Imbalanced Dataset Sampler (FIDS) to sample a formulated amount of minority label combinations on top of the training set. On a 10-fold cross validation experiment using our CHIRAL dataset (1142 manually curated molecules), our models achieved up to an accuracy of 90 % in the binary task. In the multilabel task incorporated with FIDS, the overall performance increases from 87 % to 89 % and the accuracy per label combination can attained up to a 50 % increase. Through the study of heatmaps, our work also exemplified the potential of deep neural network to make predictions based on the actual location of chirality elements.

Multi-task learning approach for modulation and wireless signal classification for 5G and beyond: Edge deployment via model compression

Future communication networks must address the scarce spectrum to accommodate extensive growth of heterogeneous wireless devices. Efforts are underway to address spectrum coexistence, enhance spectrum awareness, and bolster authentication schemes. Wireless signal recognition is becoming increasingly more significant for spectrum monitoring, spectrum management, secure communications, among others. Consequently, comprehensive spectrum awareness on the edge has the potential to serve as a key enabler for the emerging beyond 5G (fifth generation) networks. State-of-the-art studies in this domain have (i) only focused on a single task – modulation or signal (protocol) classification – which in many cases is insufficient information for a system to act on, (ii) consider either radar or communication waveforms (homogeneous waveform category), and (iii) does not address edge deployment during neural network design phase. In this work, for the first time in the wireless communication domain, we exploit the potential of deep neural networks based multi-task learning (MTL) framework to simultaneously learn modulation and signal classification tasks while considering heterogeneous wireless signals such as radar and communication waveforms in the electromagnetic spectrum. The proposed MTL architecture benefits from the mutual relation between the two tasks in improving the classification accuracy as well as the learning efficiency with a lightweight neural network model. We additionally include experimental evaluations of the model with over-the-air collected samples and demonstrate first-hand insight on model compression along with deep learning pipeline for deployment on resource-constrained edge devices. We demonstrate significant computational, memory, and accuracy improvement of the proposed model over two reference architectures. In addition to modeling a lightweight MTL model suitable for resource-constrained embedded radio platforms, we provide a comprehensive heterogeneous wireless signals dataset for public use.

Encoder-decoder neural networks for predicting future FTIR spectra - application to enzymatic protein hydrolysis.

In the process of converting food-processing by-products to value-added ingredients, fine grained control of the raw materials, enzymes and process conditions ensures the best possible yield and economic return. However, when raw material batches lack good characterization and contain high batch variation, online or at-line monitoring of the enzymatic reactions would be beneficial. We investigate the potential of deep neural networks in predicting the future state of enzymatic hydrolysis as described by Fourier-transform infrared spectra of the hydrolysates. Combined with predictions of average molecular weight, this provides a flexible and transparent tool for process monitoring and control, enabling proactive adaption of process parameters.