Stacked Network Research Articles

Cascade-refine model for cephalometric landmark detection in high-resolution orthodontic images

Landmark detection has been well developed by deep learning methods, and the cascade-connected neural network (CCNN) stands out as a widely used deep learning landmark detection method. CCNNs consist of several stacked network backbones, where the predictions of the previous network backbone are used as the input of the following one. Due to GPU memory bottlenecks, CCNNs have two limitations. First, the network backbones of CCNNs have limited numbers and depths; thus, the learning ability of CCNNs is limited. Second, CCNNs are usually trained in low-resolution images. However, the neighboring pixels in high-resolution images are usually vital for landmark detection, especially for cephalometric landmark detection. This paper interprets CCNNs as the discrete approximation of ordinary differential equations. Relying on this explanation, we further propose a novel model, called the cascade-refine model, which takes advantage of CCNNs and makes it possible to overcome the limitations of number and depth by sharing parameters among stacked network backbones. Moreover, the proposed model obeys the rule of coarse-to-fine architectures, where a global module is used to generate the coarse landmark locations, and a local module is adopted to tune the pixel error of latent landmark locations in the region of interest. The proposed model is trained in an end-to-end manner. Thus, the neighboring detailed high-resolution pixels are directly exploited for cephalometric landmark detection. The proposed cascade-refine model outperforms state-of-the-art methods on the public Automatic Cephalometric X-ray Landmark Detection Challenge 2015 dataset. We also built two private cephalometric X-ray datasets, and experimental results on both datasets demonstrate the good performance of the proposed model.

Integrating model explanations and hybrid priors into deep stacked networks for the "safe zone" prediction of acetabular cup.

Existing state-of-the-art "safe zone" prediction methods are statistics-based methods, image-matching techniques, and machine learning methods. Yet, those methods bring a tension between accuracy and interpretability. To explore the model explanations and estimator consensus for "safe zone" prediction. We collected the pelvic datasets from Orthopaedic Hospital, and a novel acetabular cup detection method is proposed for automatic ROI segmentation. Hybrid priors comprising both specific priors from data and general priors from experts are constructed. Specifically, specific priors are constructed based on the fine-tuned ResNet-101 convolutional neural networks (CNN) model, and general priors are constructed based on expert knowledge. Our method considers the model explanations and dynamic consensus through appending a SHapley Additive exPlanations (SHAP) module and a dynamic estimator stacking. The proposed method achieves an accuracy of 99.40% and an area under the curve of 0.9998. Experimental results show that our model achieves superior results to the state-of-the-art conventional ensemble classifiers and deep CNN models. This new screening model provides a new option for the "safe zone" prediction of acetabular cup.

Application of Unsupervised Transfer Technique Based on Deep Learning Model in Physical Training.

The research purpose is to study the standardization and scientizing of physical training actions. Stacking denoising auto encoder (SDAE), a BiLSTM deep network model (SDAL-DNM) (a kind of training action model), and an unsupervised transfer model are used to deeply study the action problem of physical training. Initially, the physical training action discrimination model adopted here is a combination of stacked noise reduction self-encoder and bidirectional depth network model. Then, this model can collect data for five actions in physical training and further analyze the importance of action standardization for physical training. Afterward, the SDAL-DNM implemented here fully integrates the advantages of SDAE and BiLSTM. Finally, the unsupervised transfer model adopted here is based on SDAL-DNM deep learning (DL). The movement data of the physical training crowd are collected, and then the unsupervised transfer model is trained. According to the movement characteristics of physical training, the data difference between trainers is calculated so that the actions of each trainer can be continuously adapted according to the model, and finally, the benefits of effectively distinguishing the training actions can be achieved. The research shows that before and after unsupervised learning, the average decline of the model used is 1.69%, while the average decline of extreme learning machine (ELM) is 5.5%. The conclusion is that the unsupervised transfer model can improve the discrimination accuracy of physical training actions and provide theoretical support to effectively correct mistakes in physical training actions.

Feature Fusion for Weld Defect Classification with Small Dataset

Detecting defects from weld radiography images is an important topic in the field of nondestructive testing. Many intelligent detection systems are constructed based on computer. Feature extraction is critical for constructing such system to recognize and classify the weld defects. Deep neural networks have powerful ability to learn representative features that are more sensitive to classification. However, a large number of samples are usually required. In this paper, a stacked autoencoder network is used to pretrain a deep neural network with a small dataset. We can learn the hierarchical feature from the network. In addition, two kinds of traditional manual features are extracted from the same set. These features are combined into new fusion feature vectors for classifying different weld defects. Two evaluation methods are used to test the classification performance of these features through several experiments. The results show that deep feature based on stacked autoencoder network performs better than the other features. The classification performance of fusion features is better than single feature.

Stacked deep analytic model for human activity recognition on a UCI HAR database

Background Owing to low cost and ubiquity, human activity recognition using smartphones is emerging as a trendy mobile application in diverse appliances such as assisted living, healthcare monitoring, etc. Analysing this one-dimensional time-series signal is rather challenging due to its spatial and temporal variances. Numerous deep neural networks (DNNs) are conducted to unveil deep features of complex real-world data. However, the drawback of DNNs is the un-interpretation of the network's internal logic to achieve the output. Furthermore, a huge training sample size (i.e. millions of samples) is required to ensure great performance. Methods In this work, a simpler yet effective stacked deep network, known as Stacked Discriminant Feature Learning (SDFL), is proposed to analyse inertial motion data for activity recognition. Contrary to DNNs, this deep model extracts rich features without the prerequisite of a gigantic training sample set and tenuous hyper-parameter tuning. SDFL is a stacking deep network with multiple learning modules, appearing in a serialized layout for multi-level feature learning from shallow to deeper features. In each learning module, Rayleigh coefficient optimized learning is accomplished to extort discriminant features. A subject-independent protocol is implemented where the system model (trained by data from a group of users) is used to recognize data from another group of users. Results Empirical results demonstrate that SDFL surpasses state-of-the-art methods, including DNNs like Convolutional Neural Network, Deep Belief Network, etc., with ~97% accuracy from the UCI HAR database with thousands of training samples. Additionally, the model training time of SDFL is merely a few minutes, compared with DNNs, which require hours for model training. Conclusions The supremacy of SDFL is corroborated in analysing motion data for human activity recognition requiring no GPU but only a CPU with a fast- learning rate.

DGSLSTM: Deep Gated Stacked Long Short-Term Memory Neural Network for Traffic Flow Forecasting of Transportation Networks on Big Data Environment.

Deep learning and big data techniques have become increasingly popular in traffic flow forecasting. Deep neural networks have also been applied to traffic flow forecasting. Furthermore, it is difficult to determine whether neural networks can be used for accurate traffic flow prediction. Moreover, since the network model is poorly structured and the parameter optimization technique is inappropriate, the traffic flow prediction is inaccurate because of the lack of certainty. The proposed system overcomes these problems by combining multiple simple recurrent long short-term memory (LSTM) neural networks with time traits to predict traffic flow using a deep gated stacked neural network. To deepen the model, the hidden layers have been trained using an unsupervised layer-by-layer approach. This approach provides a systematic representation of the time series data. A systematic representation of hidden layers improves the accuracy of time series forecasting by capturing information at multiple levels. Furthermore, it emphasizes the importance of model structure, random weight initialization, and hyperparameters used in stacked LSTM to enhance predictive performance. The prediction efficacy of the deep gated stacked LSTM model is compared with that of the gated recurrent unit model and the stacked autoencoder model.

MCCA-Net: Multi-color convolution and attention stacked network for Underwater image classification

Underwater images are serious problems affected by the absorption and scattering of light. At present, the existing sharpening methods can't effectively solve all underwater image degradation problems, thus it is necessary to propose a specific solution to the degradation problem. To solve the above problems, the Multi-Color Convolutional and Attentional Stacking Network (MCCA-Net) for Underwater image classification are proposed in this paper. First, an underwater image is converted to HSV and Lab color spaces and fused to achieve a refined image. Then, the attention mechanism module is used to fine the extracted image features. At last, the vertically stacked convolution module fully utilizes different levels of feature information, which realizes the fusion of convolution and attention mechanism, optimizes feature extraction and parameter reduction, and improves the classification performance of the MCCA-Net model. Extensive experiments on underwater degraded image classification show that our MCCA-Net model and method outperform other models and improve the accuracy of underwater degraded image classification. Our image fusion method can achieve 96.39% accuracy on other models, and the MCCA-Net model achieves 97.38% classification accuracy.

1D-CNN: Speech Emotion Recognition System Using a Stacked Network with Dilated CNN Features

Emotion recognition from speech data is an active and emerging area of research that plays an important role in numerous applications, such as robotics, virtual reality, behavior assessments, and emergency call centers. Recently, researchers have developed many techniques in this field in order to ensure an improvement in the accuracy by utilizing several deep learning approaches, but the recognition rate is still not convincing. Our main aim is to develop a new technique that increases the recognition rate with reasonable cost computations. In this paper, we suggested a new technique, which is a one-dimensional dilated convolutional neural network (1D-DCNN) for speech emotion recognition (SER) that utilizes the hierarchical features learning blocks (HFLBs) with a bi-directional gated recurrent unit (BiGRU). We designed a one-dimensional CNN network to enhance the speech signals, which uses a spectral analysis, and to extract the hidden patterns from the speech signals that are fed into a stacked one-dimensional dilated network that are called HFLBs. Each HFLB contains one dilated convolution layer (DCL), one batch normalization (BN), and one leaky_relu (Relu) layer in order to extract the emotional features using a hieratical correlation strategy. Furthermore, the learned emotional features are feed into a BiGRU in order to adjust the global weights and to recognize the temporal cues. The final state of the deep BiGRU is passed from a softmax classifier in order to produce the probabilities of the emotions. The proposed model was evaluated over three benchmarked datasets that included the IEMOCAP, EMO-DB, and RAVDESS, which achieved 72.75%, 91.14%, and 78.01% accuracy, respectively.

A Novel Multiscale Deep Health Indicator with Bidirectional LSTM Network for Bearing Performance Degradation Trend Prognosis

As rolling bearings are the key components in rotating machinery, bearing performance degradation directly affects machine running status. A tendency prognosis for bearing performance degradation is thus required to ensure the stability of operation. This paper proposes a novel strategy for bearing performance degradation trend prognosis, including health indicator construction techniques and a performance degradation trend prediction method. To more accurately represent the degradation trend, the multiscale deep bottleneck health indicator is proposed as a new synthesized health indicator to remove high-frequency detail signals from features, which can reduce possible fluctuations in conventional synthetic health indicators. A suitable method for selecting the statistical characteristics required for fusion is also presented to solve the problem of information redundancy that affects trend representation. In addition, a stacked autoencoder network is used for deep feature extraction of selected statistical features. A bidirectional long short-term memory network prediction model is also proposed for the prediction of degradation trend, which can make full use of historical and future information to improve prediction accuracy. Finally, experiments are carried out to verify the effectiveness of the proposed method.

Fault Diagnosis of Pitch System of Wind Turbine based on Improved Stacked Auto-Encoder Network

Fault Diagnosis of Pitch System of Wind Turbine based on Improved Stacked Auto-Encoder Network

Learning TBox With a Cascaded Anchor-Free Network for Vehicle Detection

Vehicle detection, the process of identifying vehicles as axis-aligned bounding boxes in still images, is widely used to estimate the range, time-to-collision, and motion of autonomous vehicles (AVs). Bounding boxes, while convenient, are too coarse to adapt well to vehicle shape and pose variations. In this work, we present TBox (Trapezoid & Box), a novel fine-grained representation useful for both localization and recognition that extends the bounding box by restricting the spatial extent of a vehicle to a set of keypoints and indicating semantically significant local areas using subclasses. In contrast to the previous monolithic models, we propose a cascaded anchor-free architecture to estimate the bounding box and TBox. One subnetwork uses a stacked hourglass network to detect each vehicle as a pair of corners without using anchors. Specifically, it learns corner affinity fields, enabling it to perform robust corner grouping. The other subnetwork estimates a TBox as a set of keypoints. This subnetwork utilizes the bounding box results to avoid ambiguous keypoint associations and reuses existing features to reduce the number of parameters. We also propose a multitask learning strategy for training the cascaded model that implicitly integrates the global context with local details, introducing improvements for both tasks. During testing, a refinement algorithm explicitly uses robust local keypoints to correct possible global box errors, ensuring tight geometric representations for nearby critical vehicles. The experiments show that our method outperforms existing anchor-free detectors for vehicle detection and achieves better performance on the TBox task while using a small model.

Extension of Research on Security as a Service for VMs in IaaS Platform

To satisfy security concerns including infrastructure as a service (IaaS) security framework, security service access, network anomaly detection, and virtual machine (VM) monitoring, a layered security framework is built which composes of a physical layer, a virtualization layer, and a security management layer. Then, two security service access methods are realized for various security tools from the perspective of whether security tools generate communication traffic. One without generating traffic employs the VM traffic redirection technology and the other leveraged the mechanism of multitasking process access. Moreover, a stacked LSTM-based network anomaly detection agentless method is proposed, which has advantages of a higher ratio of precision and recall. Finally, a Hypervisor-based agentless monitoring method for VMs based on dynamic code injection is proposed, which has benefits of high security of the external monitoring method and good context analysis of the internal monitoring mechanism. The experimental results demonstrate the effectiveness of the proposed protection framework and the corresponding security mechanisms, respectively.

Progressive decomposition: a method of coarse-to-fine image parsing using stacked networks

To parse images into fine-grained semantic parts, the complex elements will put it in trouble when using off-the-shelf semantic segmentation networks, because it is difficult for them to utilize the contextual information of fine-grained parts. In this paper we propose a progressive decomposition method to parse images in a coarse-to-fine manner with refined semantic classes. It consists of two aspects: stacked networks and progressive supervisions. The stacked network is achieved by stacking some segmentation layers in a segmentation network. The former segmentation module parses images at a coarser-grained level, and the result will be fed to the following one to provide effective contextual clues for the finer-grained parsing. The skip connections from shallow layers of the network to fine-grained parsing modules are also added to recover the details of small structures. For the training of the stacked networks which have coarse-to-fine outputs, a strategy of progressive supervision is proposed to merge classes in ground truth to get coarse-to-fine label maps, and then train the stacked network end-to-end with the hierarchical supervisions. The proposed framework can be injected into many advanced neural networks to improve the parsing results. Extensive evaluations on several public datasets including face parsing and human parsing well demonstrate the superiority of our method.

A Stacked Deep MEMC Network for Frame Rate Up Conversion and its Application to HEVC

Optical flows and video frame interpolation are considered as a chicken-egg problem such that one problem affects the other and vice versa. This paper presents a stack of deep networks to estimate intermediate optical flows from the very first intermediate synthesized frame and later generate the very end interpolated frame by combining the very first one and two learned intermediate optical flows based warped frames. The primary benefit is that it glues two problems into a single comprehensive framework that learns altogether by using both an analysis-by-synthesis technique for optical flow estimation and Convolutional Neural Networks (CNN) kernels-based frame synthesis. The proposed network is the first attempt to merge two previous branches of previous approaches, optical flow-based synthesis and CNN kernels-based synthesis into a comprehensive network. Experiments are carried out with various challenging datasets, all showing that the proposed network outperforms the state-of-the-art methods with significant margins for video frame interpolation and the estimated optical flows are more accurate for challenging movements. Furthermore, the proposed Motion Estimation Motion Compensation (MEMC) network shows its outstanding enhancement of the quality of compressed videos.

Construction and Reduction Methods of Web Spam Identification Index System

Background: With the rapid development of the Internet, the number of web spam has increased dramatically in recent years, which has wasted search engine storage and computing power on a massive scale. To identify the web spam effectively, the content features, link features, hidden features and quality features of web page are integrated to establish the corresponding web spam identification index system. However, the index system is highly correlation dimension. Methods: An improved method of autoencoder named stacked autoencoder neural network (SAE) is used to realize the reduction of the web spam identification index system. Results: The experiment results show that our method could reduce effectively the index of web spam and significantly improves the recognition rate in the following work. Conclusion: An autoencoder based web spam indexes reduction method is proposed in this paper. The experimental results show that it greatly reduces the temporal and spatial complexity of the future web spam detection model.

A deep feature extraction method for bearing fault diagnosis based on empirical mode decomposition and kernel function

To avoid catastrophic failures in rotating machines, it is of great significance to continuously monitor and diagnose the running state of rolling bearings. In this article, a deep feature extraction method for rolling bearing fault diagnosis based on empirical mode decomposition and kernel function is proposed. First, the vibration signals under different states of rolling bearing are decomposed by empirical mode decomposition. Second, to extract more representative high-level features, the obtained intrinsic mode functions are preprocessed with singular value decomposition to acquire singular value parameters, which are regarded as the inputs of the proposed stacked kernel sparse autoencoder network. The proposed method does not depend on prior knowledge of fault diagnosis and even does not need the signal denoising processing, simplifying the traditional process of feature extraction of rolling bearing fault diagnosis. To validate the superiority of the proposed diagnosis network, experiments and comparisons have been made as well. The achieved results demonstrated that the proposed empirical mode decomposition and stacked kernel sparse autoencoder–based diagnosis method has a superior performance in rolling bearing fault diagnosis.

MULTI-OBJECTIVE OPTIMIZATION OF A STACKED NEURAL NETWORK USED IN MODELING OF AN ELECTROLYSIS PROCESS

This study presents the results obtained from the development of an optimized stacked neural network and its application in modeling of an electrolysis process with the anodic oxidation of different organic compounds, using boron doped diamond as anode and different types of cathodes. The development of an optimized stacked neural network was performed through a hybrid evolutionary algorithm which combined the NSGA-II (Non-dominated Sorting Genetic Algorithm II) with the Quasi-Newton local search algorithm. The multi-objective optimization method provided an optimal Pareto set of non-dominated solutions representing equally good compromises between the two objectives pursued: the maximization of the generalization capacity of the stacked neural network and the minimization of its structural complexity. Achieving the two conflictual objectives meant determining the optimum value for the number of neural networks in the stack, the optimum amount of hidden neurons in each individual network from the stack, and the best weights assigned to every output of the networks composing it. The hybrid evolutionary algorithm proved to be a very suitable and easy to use tool in developing an optimized stacked neural network. The three stacked neural networks selected from the optimal Pareto front showed a great performance in modeling the electrolysis process with a correlation in the validation phase of 0.99.

Estimation of Polypropylene Melt Index through the Application of Stacked Neural Networks

Melt index is the most important parameter in determining the polypropylene grade. Since the lack of proper on-line instruments, its measurement interval and delay are both very long. This makes the quality control quite difficult. A modeling approach based on stacked neural networks is proposed to estimation the polypropylene melt index. Single neural network model generalization capability can be significantly improved by using stacked neural networks model. Proper determination of the stacking weights is essential for good stacked neural networks model performance, so determination of appropriate weights for combining individual networks using the criteria about minimization of sum of absolute prediction error is proposed. Application to real industrial data demonstrates that the polypropylene melt index can be successfully estimated using stacked neural networks. The results obtained demonstrate significant improvements in model accuracy, as a result of using stacked neural networks model, compared to using single neural network model.