Application Of Deep Neural Networks Research Articles

NOIRE-Net–a convolutional neural network for automatic classification and scaling of high-latitude ionograms

Millions of ionograms are acquired annually to monitor the ionosphere. The accumulated data contain untapped information from a range of locations, multiple solar cycles, and various geomagnetic conditions. In this study, we propose the application of deep convolutional neural networks to automatically classify and scale high-latitude ionograms. A supervised approach is implemented and the networks are trained and tested using manually analyzed oblique ionograms acquired at a receiver station located in Skibotn, Norway. The classification routine categorizes the observations based on the presence or absence of E− and F-region traces, while the scaling procedure automatically defines the E− and F-region virtual distances and maximum plasma frequencies. Overall, we conclude that deep convolutional neural networks are suitable for automatic processing of ionograms, even under auroral conditions. The networks achieve an average classification accuracy of 93% ± 4% for the E-region and 86% ± 7% for the F-region. In addition, the networks obtain scientifically useful scaling parameters with median absolute deviation values of 118 kHz ±27 kHz for the E-region maximum frequency and 105 kHz ±37 kHz for the F-region maximum O-mode frequency. Predictions of the virtual distance for the E− and F-region yield median distance deviation values of 6.1 km ± 1.7 km and 8.3 km ± 2.3 km, respectively. The developed networks may facilitate EISCAT 3D and other instruments in Fennoscandia by automatic cataloging and scaling of salient ionospheric features. This data can be used to study both long-term ionospheric trends and more transient ionospheric features, such as traveling ionospheric disturbances.

Deep Learning Voigt Profiles. I. Single-Cloud Doublets

Voigt profile (VP) decomposition of quasar absorption lines is key to studying intergalactic gas and the baryon cycle governing the formation and evolution of galaxies. The VP velocities, column densities, and Doppler b parameters inform us of the kinematic, chemical, and ionization conditions of these astrophysical environments. A drawback of traditional VP fitting is that it can be human-time intensive. With the coming next generation of large all-sky survey telescopes with multiobject high-resolution spectrographs, the time demands will significantly outstrip our resources. Deep learning pipelines hold the promise to keep pace and deliver science-digestible data products. We explore the application of deep learning convolutional neural networks (CNNs) for predicting VP-fitted parameters directly from the normalized pixel flux values in quasar absorption line profiles. A CNN was applied to 56 single-component Mg ii λ λ2796, 2803 doublet absorption line systems observed with HIRES and UVES (R = 45,000). The CNN predictions were statistically indistinct from those of a traditional VP fitter. The advantage is that, once trained, the CNN processes systems ∼105 times faster than a human expert fitting VP profiles by hand. Our pilot study shows that CNNs hold promise to perform bulk analysis of quasar absorption line systems in the future.

A novel design of layered recurrent neural networks for fractional order Caputo–Fabrizio stiff electric circuit models

Electrical engineering models often rely on complex circuit configurations that facilitate the dynamic flow of electrically charged particles within a closed conductive network. These circuits serve as essential tools for simulating and analyzing diverse electrical systems and components. This paper introduces a study on nonlinear fractional circuits modeling through the development of a stochastic neuro-computational artificial intelligent-based solver to address mathematical models governing the Fractional order Caputo–Fabrizio stiff electric circuit model (FO-CFSECM) by manipulating the knacks of layered recurrent neural networks (LRNNs) trained with Gradient-based local search algorithm (GLA). In fractional calculus, the Caputo–Fabrizio (CF) fractional order derivative (FOD) emerges as a powerful instrument, binding its capabilities to deliver remarkably accurate solutions for fractional stiff systems. The objective of this work is to exploit the numerical treatment comprehensively for the dynamics of fractal Resistor–Capacitor (RC) and fractal Resistor–Inductor (RL) circuit models by introducing the CF fractional operator. Through the application of artificial intelligence-based soft computing and advanced back-propagative deep neural networks, a deeper understanding of the behavior and distinctive characteristics inherent in these models is sought. The Levenberg–Marquardt optimizer serves as an efficient training GLA tool for learning of LRNNs weights of fractal RL/RC circuit models. The comparative studies on variants of FO-CFSECM demonstrate that LRNNs achieve an impressive mean square error (MSE) ranging from 10[Formula: see text] to 10[Formula: see text] and absolute error (AE) within 10[Formula: see text] to 10[Formula: see text]. The accuracy, reliability, and efficiency of LRNNs for solving the FO-CFSECM were further validated through MSE, AE, controlling parameters of state transitions, error histograms, and correlation measures.

Multi-scale coupled attention for visual object detection.

The application of deep neural network has achieved remarkable success in object detection. However, the network structures should be still evolved consistently and tuned finely to acquire better performance. This gears to the continuous demands on high performance in those complex scenes, where multi-scale objects to be detected are located here and there. To this end, this paper proposes a network structure called Multi-Scale Coupled Attention (MSCA) under the framework of self-attention learning with methodologies of importance assessment. Architecturally, it consists of a Multi-Scale Coupled Channel Attention (MSCCA) module, and a Multi-Scale Coupled Spatial Attention (MSCSA) module. Specifically, the MSCCA module is developed to achieve the goal of self-attention learning linearly on the multi-scale channels. In parallel, the MSCSA module is constructed to achieve this goal nonlinearly on the multi-scale spatial grids. The MSCCA and MSSCA modules can be connected together into a sequence, which can be used as a plugin to develop end-to-end learning models for object detection. Finally, our proposed network is compared on two public datasets with 13 classical or state-of-the-art models, including the Faster R-CNN, Cascade R-CNN, RetinaNet, SSD, PP-YOLO, YOLO v3, YOLO v5, YOLO v7, YOLOX, DETR, conditional DETR, UP-DETR and FP-DETR. Comparative experimental results with numerical scores, the ablation study, and the performance behaviour all demonstrate the effectiveness of our proposed model.

Machine Learning Approaches for ADHD Detection

This report explores the application of deep learning convolutional neural networks (CNNs) for analyzing brain MRI scans to assist in the diagnosis of Attention Deficit Hyperactivity Disorder (ADHD) syndrome. ADHD is a neuro developmental disorder that affects millions of individuals worldwide, characterized by difficulties in attention, hyperactivity, and impulsivity. Traditional methods of diagnosis rely on subjective assessments and behavioral observations, leading to inaccuracies and delays in treatment. Leveraging the power of deep learning CNNs for MRI analysis offers the potential for more objective and efficient diagnosis, facilitating timely intervention and personalized treatment strategies

Improving Adversarial Robustness of ECG Classification Based on Lipschitz Constraints and Channel Activation Suppression.

Deep neural networks (DNNs) are increasingly important in the medical diagnosis of electrocardiogram (ECG) signals. However, research has shown that DNNs are highly vulnerable to adversarial examples, which can be created by carefully crafted perturbations. This vulnerability can lead to potential medical accidents. This poses new challenges for the application of DNNs in the medical diagnosis of ECG signals. This paper proposes a novel network Channel Activation Suppression with Lipschitz Constraints Net (CASLCNet), which employs the Channel-wise Activation Suppressing (CAS) strategy to dynamically adjust the contribution of different channels to the class prediction and uses the 1-Lipschitz's ℓ∞ distance network as a robust classifier to reduce the impact of adversarial perturbations on the model itself in order to increase the adversarial robustness of the model. The experimental results demonstrate that CASLCNet achieves ACCrobust scores of 91.03% and 83.01% when subjected to PGD attacks on the MIT-BIH and CPSC2018 datasets, respectively, which proves that the proposed method in this paper enhances the model's adversarial robustness while maintaining a high accuracy rate.

Empirical loss weight optimization for PINN modeling laser bio-effects on human skin for the 1D heat equation

The application of deep neural networks towards solving problems in science and engineering has demonstrated encouraging results with the recent formulation of physics-informed neural networks (PINNs). Through the development of refined machine learning techniques, the high computational cost of obtaining numerical solutions for partial differential equations governing complicated physical systems can be mitigated. However, solutions are not guaranteed to be unique, and are subject to uncertainty caused by the choice of network model parameters. For critical systems with significant consequences for errors, assessing and quantifying this model uncertainty is essential. In this paper, an application of PINN for laser bio-effects with limited training data is provided for uncertainty quantification analysis. Additionally, an efficacy study is performed to investigate the impact of the relative weights of the loss components of the PINN and how the uncertainty in the predictions depends on these weights. Network ensembles are constructed to empirically investigate the diversity of solutions across an extensive sweep of hyper-parameters to determine the model that consistently reproduces a high-fidelity numerical simulation.

Application of Deep Quantum Neural Networks to Finance

The recent development of quantum computing gives us an opportunity to explore its potential applications to many fields, with the field of finance being no exception. In this paper, we apply the deep quantum neural network proposed by Beer et al. (2020) and discuss such potential in the context of simple experiments, such as learning implied volatilities and option prices. Furthermore, Greeks, such as delta and gamma, which are important measures in risk management, can be computed analytically with the neural network, and our numerical experiments show that the deep quantum neural network is a promising technique for solving such numerical problems arising in finance.

A Classification method for Insects using Data Augmentation and Deep Neural Networks

In the nature, there exists a huge number of species of insects. Insects have caused damage for human and crops. Traditional identification methods of insects require expert knowledge and time consuming. Therefore, the automatic identification and classification have been more and more necessary. In recent year, one of the efficient approaches to classify insects is the application of Deep neural networks. The paper presents the improvements for the classification of insect images. Firstly, we apply the data augmentation to improve the number of insect images. Then, we applied various deep neural networks to improve the classification accuracy of insect classification. Obtained classification accuracy of 98% on public insect datasets shows the efficiency of the proposed method.

VerSA: Versatile Systolic Array Architecture for Sparse and Dense Matrix Multiplications

A key part of modern deep neural network (DNN) applications is matrix multiplication. As DNN applications are becoming more diverse, there is a need for both dense and sparse matrix multiplications to be accelerated by hardware. However, most hardware accelerators are designed to accelerate either dense or sparse matrix multiplication. In this paper, we propose VerSA, a versatile systolic array architecture for both dense and sparse matrix multiplications. VerSA employs intermediate paths and SRAM buffers between the rows of the systolic array (SA), thereby enabling an early termination in sparse matrix multiplication with a negligible performance overhead when running dense matrix multiplication. When running sparse matrix multiplication, 256 × 256 VerSA brings performance (i.e., an inverse of execution time) improvement and energy saving by 1.21×–1.60× and 7.5–30.2%, respectively, when compared to the conventional SA. When running dense matrix multiplication, VerSA results in only a 0.52% performance overhead compared to the conventional SA.

Lightweight blueprint residual network for single image super-resolution

The application of deep convolutional neural networks (CNNs) makes the lightweight single image super-resolution (SISR) task develop rapidly in recent years. However, existing lightweight SISR networks for the terminals with limited computing resources are still not efficient enough. In order to further alleviate the model complexity while keeping remarkable reconstruction performance, in this paper, we propose a lightweight blueprint residual network (LBRN), which is a powerful CNN-based SR model while simultaneously maintaining extremely lightweight. Specifically, we design a novel lightweight blueprint residual block (LBRB) to learn the high-frequency information efficiently for lightweight SISR. The LBRB with fewer parameters consists of more efficient convolution operations to extract high-frequency information which helps recover more visual details. In addition, we propose an effective feature fusion residual block (EFFRB) which is composed of three LBRBs with convolutional kernels of different sizes. The EFFRB, which is designed with effective multi-scale receptive fields, can availably extract and fuse multi-scale high-frequency information to obtain discriminative feature, and it further improves model performance. Extensive experiments show that our LBRN achieves superior reconstruction performance quantitatively and visually compared with other lightweight state-of-the-art SR methods.

Evolution of laughter from play

ABSTRACT In this hypothesis, I discuss how laughter from physical play could have evolved to being induced via visual or even verbal stimuli, and serves as a signal to highlight incongruity that could potentially pose a threat to survival. I suggest how laughter’s induction could have negated the need for physical contact in play, evolving from its use in tickling, to tickle-misses, and to taunting, and I discuss how the application of deep learning neural networks trained on images of spectra of a variety of laughter types from a variety of individuals or even species, could be used to determine such evolutionary pathways via the use of latent space exploration.

Generating Synthetic Sperm Whale Voice Data Using StyleGAN2-ADA

The application of deep learning neural networks enables the processing of extensive volumes of data and often requires dense datasets. In certain domains, researchers encounter challenges related to the scarcity of training data, particularly in marine biology. In addition, many sounds produced by sea mammals are of interest in technical applications, e.g., underwater communication or sonar construction. Thus, generating synthetic biological sounds is an important task for understanding and studying the behavior of various animal species, especially large sea mammals, which demonstrate complex social behavior and can use hydrolocation to navigate underwater. This study is devoted to generating sperm whale vocalizations using a limited sperm whale click dataset. Our approach utilizes an augmentation technique predicated on the transformation of audio sample spectrograms, followed by the employment of the generative adversarial network StyleGAN2-ADA to generate new audio data. The results show that using the chosen augmentation method, namely mixing along the time axis, makes it possible to create fairly similar clicks of sperm whales with a maximum deviation of 2%. The generation of new clicks was reproduced on datasets using selected augmentation approaches with two neural networks: StyleGAN2-ADA and WaveGan. StyleGAN2-ADA, trained on an augmented dataset using the axis mixing approach, showed better results compared to WaveGAN.

STCS-Net: a medical image segmentation network that fully utilizes multi-scale information

In recent years, significant progress has been made in the field of medical image segmentation through the application of deep learning and neural networks. Numerous studies have focused on optimizing encoders to extract more comprehensive key information. However, the importance of decoders in directly influencing the final output of images cannot be overstated. The ability of decoders to effectively leverage diverse information and further refine crucial details is of paramount importance. This paper proposes a medical image segmentation architecture named STCS-Net. The designed decoder in STCS-Net facilitates multi-scale filtering and correction of information from the encoder, thereby enhancing the accuracy of extracting vital features. Additionally, an information enhancement module is introduced in skip connections to highlight essential features and improve the inter-layer information interaction capabilities. Comprehensive evaluations on the ISIC2016, ISIC2018, and Lung datasets validate the superiority of STCS-Net across different scenarios. Experimental results demonstrate the outstanding performance of STCS-Net on all three datasets. Comparative experiments highlight the advantages of our proposed network in terms of accuracy and parameter efficiency. Ablation studies confirm the effectiveness of the introduced decoder and skip connection module. This research introduces a novel approach to the field of medical image segmentation, providing new perspectives and solutions for future developments in medical image processing and analysis.

Physics-constrained and flow-field-message-informed graph neural network for solving unsteady compressible flows

With the increasing use of deep neural networks as surrogate models for accelerating computational simulations in mechanics, the application of artificial intelligence in computational fluid dynamics has seen renewed interest in recent years. However, the application of deep neural networks for flow simulations has mainly concentrated on relatively simple cases of incompressible flows. The strongly discontinuous structures that appear in compressible flows dominated by convection, such as shock waves, introduce significant challenges when approximating the nonlinear solutions or governing equations. In this work, we propose a novel physics-constrained, flow-field-message-informed (FFMI) graph neural network for spatiotemporal flow simulations of compressible flows involving strong discontinuities. To enhance the nonlinear approximation capability of strong discontinuities, a shock detector method is leveraged to extract the local flow-field messages. These messages are embedded into the graph representation to resolve the discontinuous solutions accurately. A new FFMI sample-and-aggregate-based message-passing layer, which aggregates the edge-weighted attributes with node features on different hop layers, is then developed to diffuse and process the flow-field messages. Furthermore, an end-to-end paradigm is established within the encoder–decoder framework to transform the extracted information from the flow field into latent knowledge about the underlying fluid mechanics. Finally, a variety of one- and two-dimensional cases involving strong shock waves are considered to demonstrate the effectiveness and generalizability of the proposed FFMI graph neural network.

Application of deep neural networks for automatic irony detection in Russian texts

The paper examines automatic methods for classifying Russian-language sentences into two classes: ironic and non-ironic. The discussed methods can be divided into three categories: classifiers based on language model embeddings, classifiers using sentiment information, and classifiers with embeddings trained to detect irony. The components of classifiers are neural networks such as BERT, RoBERTa, BiLSTM, CNN, as well as an attention mechanism and fully connected layers. The irony detection experiments were carried out using two corpora of Russian sentences: the first corpus is composed of journalistic texts from the OpenCorpora open corpus, the second corpus is an extension of the first one and is supplemented with ironic sentences from the Wiktionary resource. The best results were demonstrated by a group of classifiers based on embeddings of language models with the maximum F-measure of 0.84, achieved by a combination of RoBERTa, BiLSTM, an attention mechanism and a pair of fully connected layers in experiments on the extended corpus. In general, using the extended corpus produced results that were 2–5% higher than those of the basic corpus. The achieved results are the best for the problem under consideration in the case of the Russian language and are comparable to the best one for English.

Traffic Sign Recognition and Classification using Deep Neural Networks

Deep Neural Networks such as Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (CNNs), have been applied in various applications, including machine vision and computer vision. One of the most notable emerging applications of deep neural networks is Advanced Driver Assistance System (ADAS). In ADAS model, the deep neural networks have outperformed the manual human performance in terms of traffic sign recognition and classification. to the problem of traffic sign recognition, with promising results. This study has proposed a novel Convolutional Neural Network (CNN) architecture for recognizing and classifying traffic signs. The German Traffic Sign Recognition Benchmark dataset is used for experimental analysis. The outcome shows that the proposed model performs comparatively better. The results also indicate that the implemented CNN model with Adam optimizer and RankMax activation performs well for recognizing and classifying traffic sign images.

A Physics-Informed Low-Shot Adversarial Learning for sEMG-Based Estimation of Muscle Force and Joint Kinematics.

Muscle force and joint kinematics estimation from surface electromyography (sEMG) are essential for real-time biomechanical analysis of the dynamic interplay among neural muscle stimulation, muscle dynamics, and kinetics. Recent advances in deep neural networks (DNNs) have shown the potential to improve biomechanical analysis in a fully automated and reproducible manner. However, the small sample nature and physical interpretability of biomechanical analysis limit the applications of DNNs. This paper presents a novel physics-informed low-shot adversarial learning method for sEMG-based estimation of muscle force and joint kinematics. This method seamlessly integrates Lagrange's equation of motion and inverse dynamic muscle model into the generative adversarial network (GAN) framework for structured feature decoding and extrapolated estimation from the small sample data. Specifically, Lagrange's equation of motion is introduced into the generative model to restrain the structured decoding of the high-level features following the laws of physics. A physics-informed policy gradient is designed to improve the adversarial learning efficiency by rewarding the consistent physical representation of the extrapolated estimations and the physical references. Experimental validations are conducted on two scenarios (i.e. the walking trials and wrist motion trials). Results indicate that the estimations of the muscle forces and joint kinematics are unbiased compared to the physics-based inverse dynamics, which outperforms the selected benchmark methods, including physics-informed convolution neural network (PI-CNN), vallina generative adversarial network (GAN), and multi-layer extreme learning machine (ML-ELM).

An Innovative Inducer of Platelet Production, Isochlorogenic Acid A, Is Uncovered through the Application of Deep Neural Networks.

(1) Background: Radiation-induced thrombocytopenia (RIT) often occurs in cancer patients undergoing radiation therapy, which can result in morbidity and even death. However, a notable deficiency exists in the availability of specific drugs designed for the treatment of RIT. (2) Methods: In our pursuit of new drugs for RIT treatment, we employed three deep learning (DL) algorithms: convolutional neural network (CNN), deep neural network (DNN), and a hybrid neural network that combines the computational characteristics of the two. These algorithms construct computational models that can screen compounds for drug activity by utilizing the distinct physicochemical properties of the molecules. The best model underwent testing using a set of 10 drugs endorsed by the US Food and Drug Administration (FDA) specifically for the treatment of thrombocytopenia. (3) Results: The Hybrid CNN+DNN (HCD) model demonstrated the most effective predictive performance on the test dataset, achieving an accuracy of 98.3% and a precision of 97.0%. Both metrics surpassed the performance of the other models, and the model predicted that seven FDA drugs would exhibit activity. Isochlorogenic acid A, identified through screening the Chinese Pharmacopoeia Natural Product Library, was subsequently subjected to experimental verification. The results indicated a substantial enhancement in the differentiation and maturation of megakaryocytes (MKs), along with a notable increase in platelet production. (4) Conclusions: This underscores the potential therapeutic efficacy of isochlorogenic acid A in addressing RIT.

SecureNet: Proactive intellectual property protection and model security defense for DNNs based on backdoor learning

With the widespread application of deep neural networks (DNNs), the risk of privacy breaches against DNN models is constantly on the rise, resulting in an increasing need for intellectual property (IP) protection for such models. Although neural network watermarking techniques are widely used to safeguard the IP of DNNs, they can only achieve passive protection and cannot actively prevent unauthorized users from illicit use or embezzlement of the trained DNN models. Therefore, the development of proactive protection techniques to prevent IP infringement is imperative. To this end, we propose SecureNet, a key-based access license framework for DNN models. The proposed approach involves injecting license keys into the model through backdoor learning, enabling correct model functionality only when the appropriate license key is included in the input. To ensure the reusability of DNN models, we also propose a license key replacement algorithm. In addition, based on SecureNet, we designed defense mechanisms against adversarial attacks and backdoor attacks, respectively. Furthermore, we introduce a fine-grained authorization method that enables flexible granting of model permissions to different users. We have designed four license-key schemes with different privileges, tailored to various scenarios. We evaluated SecureNet on five benchmark datasets including MNIST, Cifar10, Cifar100, FaceScrub, and CelebA, and assessed its performance on six classic DNN models: LeNet-5, VGG16, ResNet18, ResNet101, NFNet-F5, and MobileNetV3. The results demonstrate that our approach outperforms the state-of-the-art model parameter encryption methods by at least 95% in terms of computational efficiency. Additionally, it provides effective defense against adversarial attacks and backdoor attacks without compromising the model’s overall performance.