Large Number Of Layers Research Articles

Computational Complexity Reduction of Neural Networks of Brain Tumor Image Segmentation by Introducing Fermi-Dirac Correction Functions.

Nowadays, deep learning methods with high structural complexity and flexibility inevitably lean on the computational capability of the hardware. A platform with high-performance GPUs and large amounts of memory could support neural networks having large numbers of layers and kernels. However, naively pursuing high-cost hardware would probably drag the technical development of deep learning methods. In the article, we thus establish a new preprocessing method to reduce the computational complexity of the neural networks. Inspired by the band theory of solids in physics, we map the image space into a noninteraction physical system isomorphically and then treat image voxels as particle-like clusters. Then, we reconstruct the Fermi–Dirac distribution to be a correction function for the normalization of the voxel intensity and as a filter of insignificant cluster components. The filtered clusters at the circumstance can delineate the morphological heterogeneity of the image voxels. We used the BraTS 2019 datasets and the dimensional fusion U-net for the algorithmic validation, and the proposed Fermi–Dirac correction function exhibited comparable performance to other employed preprocessing methods. By comparing to the conventional z-score normalization function and the Gamma correction function, the proposed algorithm can save at least 38% of computational time cost under a low-cost hardware architecture. Even though the correction function of global histogram equalization has the lowest computational time among the employed correction functions, the proposed Fermi–Dirac correction function exhibits better capabilities of image augmentation and segmentation.

Automatic and accurate abnormality detection from brain MR images using a novel hybrid UnetResNext-50 deep CNN model

The automatic and accurate detection and segmentation of brain tumors is a very tedious and challenging task for medical experts and radiologists. This paper proposes a hybrid deep convolutional neural network (CNN) model with a large number of layers and parameters for the automatic and accurate prediction and segmentation of brain tumors from the magnetic resonance imaging (MRI) images. The proposed model has a skip connection with cardinality which solves the problem of gradient degradation and also reduces the computational cost of Deep CNN architecture and it also improves the pixel quality at the decoder side. MRI dataset containing a total of 3929 MR images including 1373 images with tumors and 2556 images of normal type (without tumor). The dataset is preprocessed and augmented with 21 parameters before feeding the train images to the proposed models for learning. Performance metrics used for the evaluation of model efficiency are Jaccard Index, DICE score, F1-score, accuracy, precision, and recall. Our model performance is also evaluated by comparing it with the other two models UnetResNet-50 and Vanilla Unet and also with state-of-art techniques. In the post-processing stage, the scores of segmented tumor areas are calculated based on the scores of IoU and DICE and are also presented for comparison with the original images. Performance evaluation metrics show that the proposed model UnetResNext-50 shows excellent efficacy with 99.7% accuracy and a 95.73% DICE score.

MG-WFBP: Merging Gradients Wisely for Efficient Communication in Distributed Deep Learning

Distributed synchronous stochastic gradient descent has been widely used to train deep neural networks (DNNs) on computer clusters. With the increase of computational power, network communications generally limit the system scalability. Wait-free backpropagation (WFBP) is a popular solution to overlap communications with computations during the training process. In this article, we observe that many DNNs have a large number of layers with only a small amount of data to be communicated at each layer in distributed training, which could make WFBP inefficient. Based on the fact that merging some short communication tasks into a single one can reduce the overall communication time, we formulate an optimization problem to minimize the training time in pipelining communications and computations. We derive an optimal solution that can be solved efficiently without affecting the training performance. We then apply the solution to propose a distributed training algorithm named merged-gradient WFBP (MG-WFBP) and implement it in two platforms Caffe and PyTorch. Extensive experiments in three GPU clusters are conducted to verify the effectiveness of MG-WFBP. We further exploit trace-based simulations of 4 to 2048 GPUs to explore the potential scaling efficiency of MG-WFBP. Experimental results show that MG-WFBP achieves much better scaling performance than existing methods.

Enhanced Dense Space Attention Network for Super-Resolution Construction From Single Input Image

In some applications, such as surveillance and biometrics, image enlargement is required to inspect small details on the image. One of the image enlargement approaches is by using convolutional neural network (CNN)-based super-resolution construction from a single image. The first CNN-based image super-resolution algorithm is the super-resolution CNN (SRCNN) developed in 2014. Since then, many researchers have proposed several versions of CNN-based algorithms for image super-resolution to improve the accuracy or reduce the model’s running time. Currently, some algorithms still suffered from the vanishing-gradient problem and relied on a large number of layers. Thus, the motivation of this work is to reduce the vanishing-gradient problem that can improve the accuracy, and at the same time, reduce the running time of the model. In this paper, an enhanced dense space attention network (EDSAN) model is proposed to overcome the problems. The EDSAN model adopted a dense connection and residual network to utilize all the features to correlate the low-level feature and high-level feature as much as possible. Besides, implementing the convolution block attention module (CBAM) layer and multiscale block (MSB) helped reduce the number of layers required to achieve comparable results. The model is evaluated through peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM) metrics. EDSAN achieved the most significant improvement, about 1.42% when compared to the CRN model using the Set5 dataset at a scale factor of 3. Compared to the ERN model, EDSAN performed the best, with a 1.22% improvement made when using the Set5 dataset at a scale factor of 4. In terms of overall performance, EDSAN performed very well in all datasets at a scale factor of 2 and 3. In conclusion, EDSAN successfully solves the problems above, and it can be used in different applications such as biometric identification applications and real-time video applications.

Hybrid Pre-Training Strategy for Deep Denoising Neural Networks and Its Application in Machine Fault Diagnosis

Removing or at least attenuating the effect of noise is of significance for machine fault diagnosis under noisy measuring environments. In this paper, deep denoising neural networks are developed to decrease the influence of noise on the modeling accuracy of deep neural networks (DNN). Among the network, a reproducing kernel Hilbert space (RKHS) based denoising layer is designed as the first layer to reduce noise. However, due to the unstable gradients problem of the earlier layers in DNN, weights to-be-optimized in the first denoising layer are tough to update. In addition, the overfitting problem caused by multiple layers also affects the generalization ability of the model. To overcome these dilemmas, a novel hybrid pre-training strategy for deep denoising neural networks is proposed. In this strategy, motivated by the greedy algorithm, a shallow supervised network is firstly adopted to pre-train the weight of the denoising layer for alleviating the learning difficulty of the first layers in deep networks. Secondly, feature extraction layers with a large number of layers are pre-trained by an autoencoder-based unsupervised network. After that, weights of the denoising and feature extraction layers are transferred to a deep classification network, and the entire network is finally fine-tuned via supervised learning. The hybrid pre-training strategy for deep denoising neural networks is applied to machine fault diagnosis. Experiment studies verify the effectiveness of the proposed hybrid pre-training strategy in intelligent fault diagnosis of machinery under noisy measuring environments.

The Nonlinear evolutionary problem for self-stressed multilayered hyperelastic spherical bodies

The present paper studies the evolutionary problem for self-stressed multilayered spherical shells. Their stress-strain state is characterized by incompatible local finite deformations that arise due to the geometric incompatibility of the stress-free shapes of the individual layers with each other. In the considered problem, these shapes are thin-walled hollow balls that cannot be assembled into a single solid without gaps or overlaps. Such an assembly is possible only with the preliminary deformations of individual layers, which cause self-balanced stresses in them. For multilayered structures with a large number of layers, a smoothing procedure is proposed, as a result of which the piecewise continuous functions defining the preliminary deformation of the layers are replaced by continuous distributions. The reference stress-free shape of a body constructed in this way is defined within the framework of geometric continuum mechanics as a manifold with a non-Euclidean (material) connection. For the problem in question, this connection is determined by the metric tensor and its deviation from the Euclidean connection is characterized by the scalar curvature. Generalized representations for Cauchy and Piola stresses are also obtained by the methods of geometric continuum mechanics. Computations, provided for the discrete structure and body with a non-Euclidean reference shape defined by the approximation of deformation parameters, numerically illustrate the convergency of the solution for the discrete model to corresponded solution for the continuous problem if the number of layers is increasing while their total thickness is constant. In modelling it is assumed that the material of the layers is compressible, homogeneous, hyperelastic, and determined by the first-order Mooney - Rivlin elastic potential. Individual layerwise finite deformations are supposed to be centrally symmetric.

RESEARCH IN-DEPTH TRAINING OF NEURAL NETWORKS

Currently, deep learning of neural networks is one of the most popular methods for speech recognition, natural language processing, and computer vision. The article reviews the history of deep learning of neural networks and the current state in General. We consider algorithms for training neural networks used for deep training of neural networks, followed by fine-tuning using the method of back propagation of errors. Neural networks with large numbers of hidden layers, frequently occurring and disappearing gradients are very difficult to train. In this paper, we consider methods that successfully implement training of neural networks with large numbers of layers (more than one hundred) and vanishing gradients. A review of well-known libraries used for successful deep learning of neural networks is conducted.

Land-cover classification of multispectral LiDAR data using CNN with optimized hyper-parameters

Multispectral LiDAR (Light Detection And Ranging) is characterized of the completeness and consistency of its spectrum and spatial geometric data, which provides a new data source for land-cover classification. In recent years, the convolutional neural network (CNN), compared with traditional machine learning methods, has made a series of breakthroughs in image classification, object detection, and image semantic segmentation due to its stronger feature learning and feature expression abilities. However, traditional CNN models suffer from some issues, such as a large number of layers, leading to higher computational cost. To address this problem, we propose a CNN-based multi-spectral LiDAR land-cover classification framework and analyze its optimal parameters to improve classification accuracy. This framework starts with the preprocessing of multi-spectral 3D LiDAR data into 2D images. Next, a CNN model is constructed with seven fundamental functional layers, and its hyper-parameters are comprehensively discussed and optimized. The constructed CNN model with the optimized hyper-parameters was tested on the Titan multi-spectral LiDAR data, which include three wavelengths of 532 nm, 1064 nm, and 1550 nm. Extensive experiments demonstrated that the constructed CNN with the optimized hyper-parameters is feasible for multi-spectral LiDAR land-cover classification tasks. Compared with the classical CNN models (i.e., AlexNet, VGG16 and ResNet50) and our previous studies, our constructed CNN model with the optimized hyper-parameters is superior in computational performance and classification accuracies.

Knowledge Transfer for On-Device Deep Reinforcement Learning in Resource Constrained Edge Computing Systems

Deep reinforcement learning (DRL) is a promising approach for developing control policies by learning how to perform tasks. Edge devices are required to control their actions by exploiting DRL to solve tasks autonomously in various applications such as smart manufacturing and autonomous driving. However, the resource limitations of edge devices make it unfeasible for them to train their policies from scratch. It is also impractical for such an edge device to use the policy with a large number of layers and parameters, which is pre-trained by a centralized cloud infrastructure with high computational power. In this paper, we propose a method, on-device DRL with distillation (OD3), to efficiently transfer distilled knowledge of how to behave for on-device DRL in resource-constrained edge computing systems. Our proposed method makes it possible to simultaneously perform knowledge transfer and policy model compression in a single training process on edge devices with considering their limited resource budgets. The novelty of our method is to apply a knowledge distillation approach to DRL based edge device control in integrated edge cloud environments. We analyze the performance of the proposed method by implementing it on a commercial embedded system-on-module equipped with limited hardware resources. The experimental results show that 1) edge policy training with the proposed method achieves near-cloud-performance in terms of average rewards, although the size of the edge policy network is significantly smaller compared to that of the cloud policy network and 2) the training time elapsed for edge policy training with our method is reduced significantly compared to edge policy training from scratch.

Magnetization simulation of ReBCO tape stack with a large number of layers using the ANSYS A-V-A formulation

In this paper we have modeled the magnetization of a disk-shaped ReBCO tape stack by using the ANSYS A-V-A formulation based iterative algorithm method developed at Paul Scherrer Institute (PSI). The tape stack consists of 400 layers of ReBCO coated conductors from SuperOx which have a tape width of 12 mm, a thickness of 46 μm and a field angle dependent critical current density J c( B ). The model of the tape stack is created with real superconducting layers and substrates, without assuming a homogenized J c in the tape or assuming the superconducting layer as 1-D line in the tape ( T-A formulation). The computation time is reduced by using the A-V formulation in superconducting layers and the A -formulation in non-superconducting areas. Simulation studies show that the tape stack is capable of trapping a central magnetic field of 10 T after field cooling (FC) magnetization from 10 T to zero. Further studies are carried out to investigate the possibility of trapping a central magnetic field of 20 T in a tape stack by using wider or thinner ReBCO coated conductors.

Cloaking of a circular cylindrical elastic inclusion from antiplane elastic waves and resonance effects

Cloaking of a circular cylindrical elastic inclusion embedded in a homogeneous linear isotropic elastic medium from antiplane elastic waves is studied. The transformation or change-of-variables method is used to determine the material properties of the cloak and the homogenization theory of composites is used to construct a multilayered cloak consisting of many bi-material cells. The large system of algebraic equations associated with this problem is solved by using the concept of multiple scattering with wave expansion coefficient matrices. Numerical results for cloaking of an elastic inclusion and a rigid inclusion are compared with the case of a cavity. It is found that while the cloaking patterns for the three cases are similar, the major difference is that standing waves are generated in the elastic inclusion and the multilayered cloak cannot prevent the motion inside the elastic inclusion, even though the cloak seems nearly perfect. Waves can penetrate into and cause vibrations inside the elastic inclusion, where the amplitude of standing waves depend on the material properties of the inclusion but are very much reduced when compared to the case when there is no cloak. For a prescribed mass density, the displacements inside the elastic cylinder decrease as the shear modulus increases. Moreover, the cloaking of the elastic inclusion over a range of wavenumbers is also investigated. There is significant low frequency scattering even if the cloak consists of a large number of layers. When the wavenumber increases, the multilayered cloak is not effective if the cloak consists of an insufficient number of layers. Resonance effects that occur in cloaking of elastic inclusions are also discussed.

Second-harmonic generation in multilayer hexagonal boron nitride flakes

In this Letter, we report the second-harmonic generation (SHG) from thick hexagonal boron nitride (hBN) flakes with approximately 109-111 layers. The resulting effective second-order susceptibility is similar to previously reported few-layer experiments. This confirms that thick hBN flakes can serve as a platform for nonlinear optics, which is useful because thick flakes are easy to exfoliate while retaining a large size. We also show spatial second-harmonic maps revealing that SHG remains a useful tool for the characterization of the layer structure, even in the case of a large number of layers.

Optical thin-film filter design with Bragg pair constraint and a divide-and-conquer approach for reduction of the Brewster hole.

Bragg pairs are used as fundamental units to realize optimized thin-film structures. This method is efficient in designing for a large number of layers, such as in a bandpass filter or an edge filter with a relatively wide stopband or angular range. It adapts readily to conventional optimization methodology (such as genetic algorithms). In this paper, the method is first illustrated by application to a conventional high-pass coating on glass, using one set of Bragg pair materials. This demonstrated a 10% improvement over an alternative design technique. A second example is demonstrated using two sets of Bragg pair materials, designed with a more complex merit function, to increase the reflectance around the Brewster angle for p-polarized light. This demonstrated a 19% average reflection improvement across the stopband compared to a two-material design.

Digital reality: a model-based approach to supervised learning from synthetic data

Hierarchical neural networks with large numbers of layers are the state of the art for most computer vision problems including image classification, multi-object detection and semantic segmentation. While the computational demands of training such deep networks can be addressed using specialized hardware, the availability of training data in sufficient quantity and quality remains a limiting factor. Main reasons are that measurement or manual labelling are prohibitively expensive, ethical considerations can limit generating data, or a phenomenon in questions has been predicted, but not yet observed. In this position paper, we present the Digital Reality concept are a structured approach to generate training data synthetically. The central idea is to simulate measurements based on scenes that are generated by parametric models of the real world. By investigating the parameter space defined of such models, training data can be generated in a controlled way compared to data that was captured from real world situations. We propose the Digital Reality concept and demonstrate its potential in different application domains, including industrial inspection, autonomous driving, smart grid, and microscopy research in material science and engineering.

The effects of production technologies on the air permeability and crack development of cross-laminated timber

In a building envelope, the cross-laminated timber is often used as an air barrier layer. The objective of this study was to evaluate the impact of production technologies such as edge bonding, different initial moisture content of lamination and number of lamination layers (three and five) on the air permeability properties of the cross-laminated timber. Air leakage and crack area in cross-laminated timber panels were measured after the panels were conditioned in environments with different relative humidities in progressive steps from humid to dry environments (relative humidity 75% → relative humidity 43% → relative humidity 30% → relative humidity 15%). The test results showed that the five-layer specimens combined with initially drier laminations had the most considerable effect on avoiding air leakages through the panel. The greater number of layers helps to avoid any overlapping of gaps between laminations that are possible sources of air leakages. Based on the results, it is recommended to combine the technologies of using a larger number of layers together with initially drier laminations to minimise the growth of cracks on panel surfaces and avoid air leakages during the time of use.

Effective stiffness matrix method for predicting the dispersion curves in general anisotropic composites

In multi-layered composite laminates, Lamb wave equations are obtained using the transfer matrix method and global matrix method. These methods have numerical issues (missing roots or spurious roots) while solving the Lamb wave equations especially at high frequencies and for the laminates with a large number of layers. In the present work, an effective stiffness matrix method (ESM) is presented to solve the Lamb wave equations without numerical issues. The proposed ESM method offers a simple and mathematically straightforward formulation as it considers the multi-layered laminate as a single homogenous layer with effective stiffness properties. The Lamb wave equations of a single monoclinic layer are first derived by considering the displacement field in three directions and solved for obtaining dispersion curves. The proposed ESM method is then applied to various laminate configurations to test the effectiveness of the method. The different laminate configurations include quasi-isotropic, cross-ply, generally anisotropic and orthotropic laminates. The efficacy of the proposed method is established in these cases. In addition, the directional dependency of Lamb wave propagation characteristic (wave velocity) with laminate configurations is evaluated and analysed.

Deep Residual Inception Encoder-Decoder Network for Medical Imaging Synthesis.

Image synthesis is a novel solution in precision medicine for scenarios where important medical imaging is not otherwise available. The convolutional neural network (CNN) is an ideal model for this task because of its powerful learning capabilities through the large number of layers and trainable parameters. In this research, we propose a new architecture of residual inception encoder-decoder neural network (RIED-Net) to learn the nonlinear mapping between the input images and targeting output images. To evaluate the validity of the proposed approach, it is compared with two models from the literature: synthetic CT deep convolutional neural network (sCT-DCNN) and shallow CNN, using both an institutional mammogram dataset from Mayo Clinic Arizona and a public neuroimaging dataset from the Alzheimer's Disease Neuroimaging Initiative. Experimental results show that the proposed RIED-Net outperforms the two models on both datasets significantly in terms of structural similarity index, mean absolute percent error, and peak signal-to-noise ratio.

Numerical investigation into using coarse meshes to predict damage behaviors of laminated composites subjected to low-velocity impact

A high-efficiency numerical method is proposed for using coarse meshes to simulate the low-velocity impact behaviours of laminated composite structures with a relatively large number of layers. The interface strength corresponding to different levels of mesh refinement in the cohesive model is defined by deduced analytical expressions, the predictive capabilities of which are validated by measurements from double cantilever beam, end notched flexure and mixed-mode bending tests. A parameter known as saturation crack density is employed in damage evolution modelling to ensure the correct energy dissipation in coarse-mesh elements that may contain multiple cracks. Good agreement between the experimental and simulated results indicates the correctness of the proposed method for predicting impact mechanical responses and impact-induced damage. In addition, the results show that coarse-mesh models using the method proposed here not only yield consistent results, e.g., impact response curves and projected delamination area, with models with finer meshes but can also significantly improve computational efficiency.

A three-dimensional indirect boundary integral equation method for modeling elastic wave scattering in a layered half-space

A new indirect boundary integral equation method (IBIEM) is proposed in this study to solve three-dimensional (3-D) elastic wave scattering by heterogeneities in a multi-layered half-space, employing Green’s function of distributed loads on equivalent circular elements, thus avoiding the element discretization on layer interfaces. The proposed method enables the fictitious loads to be directly distributed on the surfaces of scatterer and the weak singularity to be tackled by analytical integration. Also, the radiation condition in the semi-infinite layered medium can be satisfied accurately, and the memory requirements can also be greatly reduced, especially for a large number of layers or gradient medium. The numerical accuracy was verified through comparisons with existing results and the numerical convergence was also confirmed. The results clearly demonstrate the simplicity and effectiveness of the method, and also reveals the complicated scattering characteristics in a layered half-space that are dominated by the resonant properties of the layered medium.

MI Weighted SDAE Pre-training Method for DNN and Its Application for Soft Sensing

In modern industrial process, a lot of data about the process can be obtained, and data-driven soft sensor has been widely applied. One of them is deep neural networks (DNN) based soft sensor. Due to the large number of layers in DNN, the parameters of the networks are difficult to converge to a good value by back-propagation directly. This paper presented a novel mutual information (MI) weighted stacked denoising autoencoder (SDAE) method to find initial values of DNN. Compared with the traditional pre-training method like SDAE, the proposed method can obtain the parameters of DNN that can get features which have more MI with outputs. This method was applied to build soft sensor of debutanizer column. Experiment results indicate that when the dimension of deep features is selected properly, DNN pre-trained by MI weighted SDAE is more accurate than DNN pre-trained by SDAE and DNN without pre-training in terms of RMSE and r, thus verifying the feasibility and effectiveness of the proposed method.