Residual resampling-based physics-informed neural network for neutron diffusion equations

Residual neural network genomic selection is the first GS algorithm to reach 35 layers, and its prediction accuracy surpasses previous algorithms. With the decrease in DNA sequencing costs and the development of deep learning, phenotype prediction accuracy by genomic selection (GS) continues to improve. Residual networks, a widely validated deep learning technique, are introduced to deep learning for GS. Since each locus has a different weighted impact on the phenotype, strided convolutions are more suitable for GS problems than pooling layers. Through the above technological innovations, we propose a GS deep learning algorithm, residual neural network for genomic selection (ResGS). ResGS is the first neural network to reach 35 layers in GS. In 15 cases from four public data, the prediction accuracy of ResGS is higher than that of ridge-regression best linear unbiased prediction, support vector regression, random forest, gradient boosting regressor, and deep neural network genomic prediction in most cases. ResGS performs well in dealing with gene-environment interaction. Phenotypes from other environments are imported into ResGS along with genetic data. The prediction results are much better than just providing genetic data as input, which demonstrates the effectiveness of GS multi-modal learning. Standard deviation is recommended as an auxiliary GS evaluation metric, which could improve the distribution of predicted results. Deep learning for GS, such as ResGS, is becoming more accurate in phenotype prediction.

Determining the presence of tumor in biopsies and the decision-making during resections is often dependent on intraoperative rapid frozen-section histopathology. Recently, stimulated Raman scattering microscopy has been introduced to rapidly generate digital hematoxylin-and-eosin-stained-like images (stimulated Raman histology) for intraoperative analysis. To enable intraoperative prediction of tumor presence, we aimed to develop a new deep residual convolutional neural network in an automated pipeline and tested its validity. In a monocentric prospective clinical study with 94 patients undergoing biopsy, brain or spinal tumor resection, Stimulated Raman histology images of intraoperative tissue samples were obtained using a fiber-laser-based stimulated Raman scattering microscope. A residual network was established and trained in ResNetV50 to predict three classes for each image: (1) tumor, (2) non-tumor, and (3) low-quality. The residual network was validated on images obtained in three small random areas within the tissue samples and were blindly independently reviewed by a neuropathologist as ground truth. 402 images derived from 132 tissue samples were analyzed representing the entire spectrum of neurooncological surgery. The automated workflow took in a mean of 240 s per case, and the residual network correctly classified tumor (305/326), non-tumorous tissue (49/67), and low-quality (6/9) images with an inter-rater agreement of 89.6% (κ = 0.671). An excellent internal consistency was found among the random areas with 90.2% (Cα = 0.942) accuracy. In conclusion, the novel stimulated Raman histology-based residual network can reliably detect the microscopic presence of tumor and differentiate from non-tumorous brain tissue in resection and biopsy samples within 4 min and may pave a promising way for an alternative rapid intraoperative histopathological decision-making tool.

An adaptive optical wavefront recovery method based on a residual attention network is proposed for the degradation of an Orbital Angular Momentum multiplexing communication system performance caused by atmospheric turbulence in free-space optical communication. To prevent the degeneration phenomenon of neural networks, the residual network is used as the backbone network, and a multi-scale residual hybrid attention network is constructed. Distributed feature extraction by convolutional kernels at different scales is used to enhance the network’s ability to represent light intensity image features. The attention mechanism is used to improve the recognition rate of the network for broken light spot features. The network loss function is designed by combining realistic evaluation indexes so as to obtain Zernike coefficients that match the actual wavefront aberration. Simulation experiments are carried out for different atmospheric turbulence intensity conditions, and the results show that the residual attention network can reconstruct the turbulent phase quickly and accurately. The peaks to valleys of the recovered residual aberrations were between 0.1 and 0.3 rad, and the root means square was between 0.02 and 0.12 rad. The results obtained by the residual attention network are better than those of the conventional network at different SNRs.

The detection of gravitational waves is considered to be one of the most magnificent discoveries of the century. Due to the high computational cost of matched filtering pipeline, there is a hunt for an alternative powerful system. I present the use of 1D residual neural network for detection of gravitational waves. Residual networks have transformed many fields like image classification, face recognition and object detection with their robust structure. With increase in sensitivity of LIGO detectors we expect many more sources of gravitational waves in the universe to be detected. However, deep learning networks are trained only once. When used for classification task, deep neural networks are trained to predict only a fixed number of classes. Therefore, when a new type of gravitational wave is to be detected, this turns out to be a drawback of deep learning. Shallow neural networks can be used to learn data with simple patterns but fail to give good results with increase in complexity of data. Remodelling the neural network with detection of each new type of GW is highly infeasible. In this letter, I also discuss ways to reduce the time required to adapt to such changes in detection of gravitational waves for deep learning methods. Primarily, I aim to create a custom residual neural network for 1-dimensional time series inputs, which can learn a ton of features from dataset without giving up on increasing the number of classes or increasing the complexity of data. I use two of the classes of binary coalesce signals (Binary Black Hole Merger and Binary Neutron Star Merger signals) detected by LIGO to check the performance of residual structure on gravitational waves detection.

Recent developments in deep learning methods have greatly influenced the performances of speech recognition systems. In a Hidden Markov model-Deep neural network (HMM-DNN) based speech recognition system, DNNs have been employed to model senones (context dependent states of HMM), where HMMs capture the temporal relations among senones. Due to the use of more deeper networks significant improvement in the performances has been observed and developing deep learning methods to train more deeper architectures has gained a lot of scientific interest. Optimizing a deeper network is more complex task than to optimize a less deeper network, but recently residual network have exhibited a capability to train a very deep neural network architectures and are not prone to vanishing/exploding gradient problems. In this work, the effectiveness of residual networks have been explored for of speech recognition. Along with the depth of the residual network, the criticality of width of the residual network has also been studied. It has been observed that at higher depth, width of the networks is also a crucial parameter for attaining significant improvements. A 14-hour subset of WSJ corpus is used for training the speech recognition systems, it has been observed that the residual networks have shown much ease in convergence even with a depth much higher than the deep neural network. In this work, using residual networks an absolute reduction of 0.4 in WER error rates (8% reduction in the relative error) is attained compared to the best performing deep neural network.

Surrogate modeling for neutron diffusion problems based on conservative physics-informed neural networks with boundary conditions enforcement

Machine learning and computer vision have driven many of the greatest advances in the modeling of Deep Convolutional Neural Networks (DCNNs). Nowadays, most of the research has been focused on improving recognition accuracy with better DCNN models and learning approaches. The recurrent convolutional approach is not applied very much, other than in a few DCNN architectures. On the other hand, Inception-v4 and Residual networks have promptly become popular among computer the vision community. In this paper, we introduce a new DCNN model called the Inception Recurrent Residual Convolutional Neural Network (IRRCNN), which utilizes the power of the Recurrent Convolutional Neural Network (RCNN), the Inception network, and the Residual network. This approach improves the recognition accuracy of the Inception-residual network with same number of network parameters. In addition, this proposed architecture generalizes the Inception network, the RCNN, and the Residual network with significantly improved training accuracy. We have empirically evaluated the performance of the IRRCNN model on different benchmarks including CIFAR-10, CIFAR-100, TinyImageNet-200, and CU3D-100. The experimental results show higher recognition accuracy against most of the popular DCNN models including the RCNN. We have also investigated the performance of the IRRCNN approach against the Equivalent Inception Network (EIN) and the Equivalent Inception Residual Network (EIRN) counterpart on the CIFAR-100 dataset. We report around 4.53, 4.49 and 3.56% improvement in classification accuracy compared with the RCNN, EIN, and EIRN on the CIFAR-100 dataset respectively. Furthermore, the experiment has been conducted on the TinyImageNet-200 and CU3D-100 datasets where the IRRCNN provides better testing accuracy compared to the Inception Recurrent CNN, the EIN, the EIRN, Inception-v3, and Wide Residual Networks.

The Deep Convolutional Neural Network (DCNN) is one of the most powerful and successful deep learning approaches. DCNNs have already provided superior performance in different modalities of medical imaging including breast cancer classification, segmentation, and detection. Breast cancer is one of the most common and dangerous cancers impacting women worldwide. In this paper, we have proposed a method for breast cancer classification with the Inception Recurrent Residual Convolutional Neural Network (IRRCNN) model. The IRRCNN is a powerful DCNN model that combines the strength of the Inception Network (Inception-v4), the Residual Network (ResNet), and the Recurrent Convolutional Neural Network (RCNN). The IRRCNN shows superior performance against equivalent Inception Networks, Residual Networks, and RCNNs for object recognition tasks. In this paper, the IRRCNN approach is applied for breast cancer classification on two publicly available datasets including BreakHis and Breast Cancer (BC) classification challenge 2015. The experimental results are compared against the existing machine learning and deep learning-based approaches with respect to image-based, patch-based, image-level, and patient-level classification. The IRRCNN model provides superior classification performance in terms of sensitivity, area under the curve (AUC), the ROC curve, and global accuracy compared to existing approaches for both datasets.

UAV remote sensing has been widely used in emergency rescue, disaster relief, environmental monitoring, urban planning, and so on. Image recognition and image location in environmental monitoring has become an academic hotspot in the field of computer vision. Convolution neural network model is the most commonly used image processing model. Compared with the traditional artificial neural network model, convolution neural network has more hidden layers. Its unique convolution and pooling operations have higher efficiency in image processing. It has incomparable advantages in image recognition and location and other forms of two-dimensional graphics tasks. As a new deformation of convolution neural network, residual neural network aims to make convolution layer learn a kind of residual instead of a direct learning goal. After analyzing the characteristics of CNN model for image feature representation and residual network, a residual network model is built. The UAV remote sensing system is selected as the platform to acquire image data, and the problem of image recognition based on residual neural network is studied, which is verified by experiment simulation and precision analysis. Finally, the problems and experiences in the process of learning and designing are discussed, and the future improvements in the field of image target location and recognition are prospected.

One of the challenges in computer vision is age classification. There have been many methods used to classify someone age from the image of their faces. Convolutional neural network (CNN) gives a high accuracy but it cannot be used on many layers. Therefore, a residual technique is applied on convolutional neural network then named residual neural network. In this paper, some Residual Networks are applied to develop an age classification with face image using the Adience dataset that has 19,370 face images from 2,284 individuals grouped into eight categories: 0-2, 4-6, 8-13, 15-20, 25-32, 38-43, 48-53, and 60-100 years. Three techniques: cyclical learning rate, data augmentation, and transfer learning are observed. Six training scenarios are performed to select the best model. Experimental results show that Resnet34 is the best model with an average F1 score of 0.792 that is achieved by data augmentation, transfer learning, and trained on the image with size 224 x 224 pixels.

Porosity is an integral part of reservoir evaluation,but in thefield of reservoir prediction, due to the complex nonlinear relationshipbetween logging parameters and porosity, linear models cannot accuratelypredict porosity. Therefore, this paper uses machine learning methodsthat can better handle the relationship between nonlinear loggingparameters and porosity to predict porosity. In this paper, loggingdata from Tarim Oilfield are selected for model testing, and thereis a nonlinear relationship between these parameters and porosity.First, the data features of the logging parameters are extracted bythe residual network, which uses the “hop connections”method to transform the original data closer to the target variable.In addition, the residual blocks inside the residual network use jumpconnections, which alleviates the gradient vanishing problem causedby increasing depth in deep neural networks. The dynamic nature ofdata would necessitate LSTM in the first place. Then, a bidirectionallong short-term network (BiLSTM) is used to predict the porosity ofthe extracted logging data features. Among them, the BiLSTM is composedof two independent reverse LSTMs, which can better solve the nonlinearprediction problem. In order to further improve the accuracy of themodel, this paper introduces an attention mechanism to learn by weightingeach of the inputs in proportion to their impact on the porosity.The experimental results also show that the data features extractedby the residual neural network can be better used as the input ofthe BiLSTM model.

Emotion recognition has become a research hotspot due to the rapid development of machine learning and neuroscience. One of the most challenging tasks in the Brain Computer Interface (BCI) is to recognize human emotions by electroencephalography (EEG) signals. Motivated by the excellent performance of deep learning approaches in recognition tasks, we proposed an All-Convolution Residual Neural Network (ACRNN), which is a hybrid neural network that combines convolution neural network (CNN) and residual network (ResNet). The ACRNN solves the problem of information loss between convolution layer and full connection layer to some extent, and the time hardly increase. Meanwhile, instead of pooling layer, we increased the convolution step to reduce the size of the feature map, so there was no pooling layer in ACRNN. We conducted extensive experiments on the DEAP dataset to demonstrate the performance of the emotional recognition of the ACRNN. The experimental results demonstrate that the proposed method achieved an excellent performance with a recognition accuracy of 92.46% and 91.68% on arousal and valence classification task. It was verified that the ACRNN for emotion recognition is effective.

ResNet and its application to medical image processing: Research progress and challenges

Deep residual convolutional neural network (CNN) has recently achieved great success in image super-resolution (SR). Because residual learning accelerates convergence rate and eases the difficulty for reconstructing high-resolution (HR) image, these CNN models can achieve higher peak signal to noise ratio (PSNR) values with lower training cost. However, residual image used in present residual network still contains much high frequency information, which increases learning burden and limits learning ability of residual network. Moreover, training a very deep network faces many obstacles and costs too much time. In this paper, we propose a multiple residual learning network (MRLN), which not only further simplifies information complexity of residual image and improves the accuracy of residual network, but also obviously reduces time cost for training a very deep CNN. In MRLN, we use a shallow network formed by 30-layer convolutional layers as basic model and train it for multiple times. The output of previous basic model is used as the HR input of the next one. In this way, an extremely large CNN is converted into a series connection of shallow networks. Fig. 1 shows PSNR of recent state-of-the-art CNN models for scale factor 2 on Set5, our method performs better than other methods and set a new level for SR.

As a popular renewable energy generation technology, wind turbine system has become a critical enabler for building the sustainable cyber‐physical system (CPS). The main shaft bearing is an important part of the wind turbine CPS and often runs under variable working conditions. Thus, the reliable bearing diagnosis method can timely discover the main shaft bearing fault, which reduces the maintenance cost of wind turbines. Inspired by the idea of domain adaptation, we combined domain adversarial neural network and residual network and proposed a novel deep domain adversarial residual neural network (DDA‐RNN) for diagnosing bearing fault and improving model performance on the unlabeled dataset. This proposed software and hardware co‐design method was evaluated by our bearing dataset, which was collected from two wind turbine CPSs from Sanmenxia in Henan Province. Besides, F1 score and accuracy are served as model metrics, which reflect the diagnosis performance. Compared with other methods, the experimental results show that DDA‐RNN can improve model performance. Meanwhile, DDA‐RNN extracts diagnosis knowledge from labeled dataset and improves the model performance on the unlabeled dataset under different working condition. Therefore, the proposed method can be potentially used to benefit many practical scenarios in the future.