Deep Neural Network Structure Research Articles

Multimodal Connectivity‐Guided Glioma Segmentation From Magnetic Resonance Images via Cascaded 3D Residual U‐Net

ABSTRACTGlioma is a type of brain tumor with a high mortality rate. Magnetic resonance imaging (MRI) is commonly used for examination, and the accurate segmentation of tumor regions from MR images is essential to computer‐aided diagnosis. However, due to the intrinsic heterogeneity of brain glioma, precise segmentation is very challenging, especially for tumor subregions. This article proposed a two‐stage cascaded method for brain tumor segmentation that considers the hierarchical structure of the target tumor subregions. The first stage aims to identify the whole tumor (WT) from the background area; and the second stage aims to achieve fine‐grained segmentation of the subregions, including enhanced tumor (ET) region and tumor core (TC) region. Both stages apply a deep neural network structure combining modified 3D U‐Net with a residual connection scheme to tumor region and subregion segmentation. Moreover, in the training phase, the 3D masks generation of subregions with potential incomplete connectivity are guided by the completely connected regions. Experiments were performed to evaluate the performance of the methods on both area and boundary accuracy. The average dice score of the WT, TC, and ET regions on BraTS 2020 dataset is 0.9168, 0.0.8992, 0.8489, and the Hausdorff distance is 6.021, 9.203, 12.171, respectively. The proposed method outperforms current works, especially in segmenting fine‐grained tumor subregions.

Hybrid wind speed optimization forecasting system based on linear and nonlinear deep neural network structure and data preprocessing fusion

Wind speed time series forecasting has been widely used in wind power generation. However, the nonlinear and non-stationary characteristics of wind speed make accurate wind speed forecasting a difficult task. In recent years, the rapid development of artificial intelligence and machine learning technology provides a new solution to the problem of wind speed forecasting. Combining the advance of artificial intelligence and data analysis strategy, this paper proposes a wind speed forecasting system based on multi-model fusion and integrated learning. Considering the differences in data observation and training principles of various algorithms and exploiting the advantages of each model, a wind speed forecasting system with multiple machine learning algorithms embedded in integrated learning is constructed, which includes three modules: data preprocessing, optimization and forecasting. The data preprocessing module can conduct quantitative analysis through input data decomposition and feature extraction, and the combination of multi-objective intelligent optimization algorithm and combined forecasting method can effectively forecast the wind speed time series. The validity of the algorithm is verified using the data of Shandong wind farm in China. The forecasting results show that compared with the traditional single model forecasting, the proposed integrated wind speed forecasting system based on the fusion of multiple models has higher forecasting accuracy.

Enhanced Speech Processing for Air Traffic Control Using an Optimized Generative Adversarial Network

In order to improve the quality of voice calls during air traffic control, an improved SEGAN air traffic control speech enhancement algorithm is proposed. Aiming at the problem that the traditional speech enhancement algorithm based on generative adversarial network (SEGAN) is drowned under low signal-to-noise ratio conditions, a multi-stage, multi-mapping, multi-dimensional output generator and a multi-scale, multi-discriminator network model are proposed based on the SEGAN network model. First, the speech semantic features are extracted based on the deep neural network structure to complete the semantic segmentation of air traffic control speech. Secondly, multiple generators are set to further optimize the speech signal. Then, a downsampling module is added to the convolution layer to improve the model's utilization of speech information and reduce the loss of speech information. Finally, multi-scale, multiple discriminators are used to learn the distribution law and information of speech samples in multiple directions. The results show that under low signal-to-noise ratio conditions, the improved SEGAN model improves the short-term objective intelligibility (STOI) and the perceptual evaluation of speech quality (PESQ) by and respectively , which can quickly and effectively perform air traffic control speech enhancement and provide preparation for subsequent air traffic control speech recognition.

A novel deep neural network structure for software fault prediction

A novel deep neural network structure for software fault prediction

Determination of heat transfer characteristics of tube bundles over heating plate by machine learning

In this study machine learning is used with the aid of a deep neural network algorithm to predict convective heat transfer characteristics for inline and staggered tube bundles, and correlation equations for Nusselt number and friction factor are derived. Machine learning algorithm's data were obtained from experimental work for various transverse pitch of the tube bundles, longitudinal pitch of the tube bundles and Reynolds number. 276 experimental data points were taken for both inline and staggered tube bundles. However, considering that the data obtained from the experimental study may be insufficient for training, a two-step data augmentation method and retraining with cross-validation was used to prevent data deficiency in the deep neural network structure. Thus, the unseen data in the experimental work were also predicted. The coefficient of determination for the DNN model predictions was obtained greater than 0.96. One correlation equation for Nusselt Number and three correlation equations for friction factor were proposed from the augmented data with machine learning. The R2 values of the correlation equations varied between 89 % and 99 %. As a result, machine learning methods successfully applied to predict the Nusselt number and friction factor of tube bundles consistent with the experimental data.

Weighted Expectile Regression Neural Networks for Right Censored Data.

As a favorable alternative to the censored quantile regression, censored expectile regression has been popular in survival analysis due to its flexibility in modeling the heterogeneous effect of covariates. The existing weighted expectile regression (WER) method assumes that the censoring variable and covariates are independent, and that the covariates effects has a global linear structure. However, these two assumptions are too restrictive to capture the complex and nonlinear pattern of the underlying covariates effects. In this article, we developed a novel weighted expectile regression neural networks (WERNN) method by incorporating the deep neural network structure into the censored expectile regression framework. To handle the random censoring, we employ the inverse probability of censoring weighting (IPCW) technique in the expectile loss function. The proposed WERNN method is flexible enough to fit nonlinear patterns and therefore achieves more accurate prediction performance than the existing WER method for right censored data. Our findings are supported by extensive Monte Carlo simulation studies and a real data application.

Motion coprime array-based DOA estimation considering phase disturbance of sensor array

To address the phase disturbance issue faced by sensor arrays in practical applications, a cascaded deep convolutional neural network structure is proposed to achieve direction-of-arrival (DOA) estimation for motion coprime arrays. Firstly, the synthesized covariance matrix obtained after motion is inputted into the first-level network for estimating the phase disturbance matrix. Then, we analyze the impact of phase perturbation on the synthesized covariance matrix and utilize the estimated disturbance phase to obtain an undisturbed synthesized covariance matrix. Finally, after phase compensation, the synthesized covariance matrix performs DOA estimation through the second-level network. Furthermore, to acquire three times the virtual unique lags of the coprime array, the synthesis condition about moving distance and unique lags is derived. The proposed method is shown to be effective and superior through the experiment results.

Temperature compensation method for impedance signals of bolt loosening under temperature varying conditions using TransUnet

Abstract Piezoelectric transducer based electro-mechanical impedance technology is an effective and reliable detection method for bolt structure loosening under external load. However, temperature change usually causes some non-loosening factors to change the impedance spectrum of bolt structure. In order to reduce the misjudgment of loosening damage caused by temperature change, it is necessary to construct a temperature compensation model for impedance spectrum. In this paper, the convolutional structure in U-net and Transformer are effectively combined to form a TransUnet deep neural network structure suitable for input and output of one-dimensional data. Using impedance data between 10 °C and 50 °C and temperature as input to the network. After convolution operation, the convolutional block attention module is embedded in the U-net to optimize the encoder transmission characteristics and enhance the performance of the skip connection in the traditional U-shaped structure. The temperature compensation rate (TCR) is defined to measure the effect of temperature compensation. Then the lightweight convolutional neural network structure is used to recognize the bolt loosening damage of the compensated impedance signal. The generalization ability of the TransUnet was tested using impedance data that is not in the training dataset. The results show that the TransUnet proposed in the paper can realize the temperature compensation of multi-peak impedance signals. The TCR and recognition accuracy of bolt loose damage reaches reach 0.003 and 95.7%, respectively, which is 11% higher than that without temperature compensation. At 60 °C, the TCR and the identification accuracy of loosening damage can still reach 0.0055 and 90.2%, respectively, which show that the TransUnet has strong generalization ability.

Exploring the impact of network depth on 3D U-Net-based dose prediction for cervical cancer radiotherapy.

The 3D U-Net deep neural network structure is widely employed for dose prediction in radiotherapy. However, the attention to the network depth and its impact on the accuracy and robustness of dose prediction remains inadequate. 92 cervical cancer patients who underwent Volumetric Modulated Arc Therapy (VMAT) are geometrically augmented to investigate the effects of network depth on dose prediction by training and testing three different 3D U-Net structures with depths of 3, 4, and 5. For planning target volume (PTV), the differences between predicted and true values of D98, D99, and Homogeneity were statistically 1.00 ± 0.23, 0.32 ± 0.72, and -0.02 ± 0.02 for the model with a depth of 5, respectively. Compared to the other two models, these parameters were also better. For most of the organs at risk, the mean and maximum differences between the predicted values and the true values for the model with a depth of 5 were better than for the other two models. The results reveal that the network model with a depth of 5 exhibits superior performance, albeit at the expense of the longest training time and maximum computational memory in the three models. A small server with two NVIDIA GeForce RTX 3090 GPUs with 24 G of memory was employed for this training. For the 3D U-Net model with a depth of more than 5 cannot be supported due to insufficient training memory, the 3D U-Net neural network with a depth of 5 is the commonly used and optimal choice for small servers.

Unambiguous and High-Fidelity Backdoor Watermarking for Deep Neural Networks.

The unprecedented success of deep learning could not be achieved without the synergy of big data, computing power, and human knowledge, among which none is free. This calls for the copyright protection of deep neural networks (DNNs), which has been tackled via DNN watermarking. Due to the special structure of DNNs, backdoor watermarks have been one of the popular solutions. In this article, we first present a big picture of DNN watermarking scenarios with rigorous definitions unifying the black- and white-box concepts across watermark embedding, attack, and verification phases. Then, from the perspective of data diversity, especially adversarial and open set examples overlooked in the existing works, we rigorously reveal the vulnerability of backdoor watermarks against black-box ambiguity attacks. To solve this problem, we propose an unambiguous backdoor watermarking scheme via the design of deterministically dependent trigger samples and labels, showing that the cost of ambiguity attacks will increase from the existing linear complexity to exponential complexity. Furthermore, noting that the existing definition of backdoor fidelity is solely concerned with classification accuracy, we propose to more rigorously evaluate fidelity via examining training data feature distributions and decision boundaries before and after backdoor embedding. Incorporating the proposed prototype guided regularizer (PGR) and fine-tune all layers (FTAL) strategy, we show that backdoor fidelity can be substantially improved. Experimental results using two versions of the basic ResNet18, advanced wide residual network (WRN28_10) and EfficientNet-B0, on MNIST, CIFAR-10, CIFAR-100, and FOOD-101 classification tasks, respectively, illustrate the advantages of the proposed method.

Multimodal functional deep learning for multiomics data.

With rapidly evolving high-throughput technologies and consistently decreasing costs, collecting multimodal omics data in large-scale studies has become feasible. Although studying multiomics provides a new comprehensive approach in understanding the complex biological mechanisms of human diseases, the high dimensionality of omics data and the complexity of the interactions among various omics levels in contributing to disease phenotypes present tremendous analytical challenges. There is a great need of novel analytical methods to address these challenges and to facilitate multiomics analyses. In this paper, we propose a multimodal functional deep learning (MFDL) method for the analysis of high-dimensional multiomics data. The MFDL method models the complex relationships between multiomics variants and disease phenotypes through the hierarchical structure of deep neural networks and handles high-dimensional omics data using the functional data analysis technique. Furthermore, MFDL leverages the structure of the multimodal model to capture interactions between different types of omics data. Through simulation studies and real-data applications, we demonstrate the advantages of MFDL in terms of prediction accuracy and its robustness to the high dimensionality and noise within the data.

A deep learning approach for solving the stationary compositional two-phase equilibrium problems

In this paper, we propose and investigate a deep neural network approach for solving the stationary compositional two-phase equilibrium problems in porous media. A recent approach is the unified formulation advocated by Lauser et al. (2011) which contains the complementarity conditions. The advantage of this formulation lies in its potential to handle the appearance and disappearance of phases automatically. To solve numerically the system of equations, a new strategy called NPIPM (NonParametric Interior-Point Method) is proposed by Vu et al. (2021). However, the method still has some disadvantages preventing the convergence to a solution. Taking inspiration from the work of Raissi et al. (2019) for the study of nonlinear partial differential equations using Physics-Informed Neural Networks (PINN), we realize that we can apply a deep neural network to obtain the solution of the stationary compositional model containing gas and liquid. We design a deep neural network structure and define a new loss function for the problem. The effectiveness of the proposed method is validated through numerical results obtained from a series of systems with increasing components. This machine learning approach provides a promising idea to solve the curse of dimensionality issue in the sense of the increasing number of components in stationary problems.

Greedy deep stochastic configuration networks ensemble with boosting negative correlation learning

Deep stochastic configuration networks (DSCNs) employ data-dependent supervision mechanism to randomly assign node parameters and incrementally construct the deep neural network structure, thereby ensuring the model's universal approximation property. To build a random neural networks ensemble model with better generalization performance, we propose a novel greedy deep stochastic configuration networks ensemble model based on boosting negative correlation learning, termed as GDSCNE. Firstly, greedy optimization strategy based on inequality constraints is utilized to generate random parameters of base components with multi-layer architecture, which can accelerate the decline of network residuals when configuring a new node. Additionally, boosting negative correlation learning framework is presented for the base components ensemble process, which uses least square approach with negative correlation learning penalty term to update the ensemble output weights for each base component, subsequently, boosting method is applied to construct a stronger ensemble model by adaptive weighting through the results of base components. Finally, we evaluated GDSCNE on the popular regression benchmark datasets from the KEEL, experimental results demonstrate that GDSCNE outperforms state-of-the-art random learning algorithms in terms of regression accuracy and generalization performance across several regression datasets with varying sizes.

A Deep Learning-Based Emergency Alert Wake-Up Signal Detection Method for the UHD Broadcasting System.

With the increasing frequency and severity of disasters and accidents, there is a growing need for efficient emergency alert systems. The ultra-high definition (UHD) broadcasting service based on Advanced Television Systems Committee (ATSC) 3.0, a leading terrestrial digital broadcasting system, offers such capabilities, including a wake-up function for minimizing damage through early alerts. In case of a disaster situation, the emergency alert wake-up signal is transmitted, allowing UHD TVs to be activated, enabling individuals to receive emergency alerts and access emergency broadcasting content. However, conventional methods for detecting the bootstrap signal, essential for this function, typically require an ATSC 3.0 demodulator. In this paper, we propose a novel deep learning-based method capable of detecting an emergency wake-up signal without the need for an ATSC 3.0. The proposed method leverages deep learning techniques, specifically a deep neural network (DNN) structure for bootstrap detection and a convolutional neural network (CNN) structure for wake-up signal demodulation and to detect the bootstrap and 2 bit emergency alert wake-up signal. Specifically, our method eliminates the need for Fast Fourier Transform (FFT), frequency synchronization, and interleaving processes typically required by a demodulator. By applying a deep learning in the time domain, we simplify the detection process, allowing for the detection of an emergency alert signal without the full suite of demodulator components required for ATSC 3.0. Furthermore, we have verified the performance of the deep learning-based method using ATSC 3.0-based RF signals and a commercial Software-Defined Radio (SDR) platform in a real environment.

InceptCurves: curve reconstruction using an inception network

Curve reconstruction is a fundamental task in many visual computing applications. In this paper, a data-driven approach for curve reconstruction is proposed. We present an inception layered deep neural network structure, capable of learning simultaneously the number of control points and their positions in order to reconstruct the curve. To train the network, a large set of general synthetic data is generated. The reconstructed uniform B-spline closely approximates any arbitrary input curve, with or without intersections. Because the network predicts the number of control points required for the B-spline reconstruction, redundancy is reduced in the curve representation. We demonstrate our approach on various examples.

A new lightweight network for efficient UAV object detection

Optimizing the structure of deep neural networks is essential in many applications. Especially in the object detection tasks of Unmanned Aerial Vehicles. Due to the constraints of the onboard platform, a more efficient network is required to meet practical demands. Nevertheless, existing lightweight detection networks exhibit excessive redundant computations and may yield in a certain level of accuracy loss. To address these issues, this paper proposes a new lightweight network structure named Cross-Stage Partially Deformable Network (CSPDNet). The initial proposal consists of a Deformable Separable Convolution Block (DSCBlock), separating feature channels, greatly reducing the computational load of convolution, and applying adaptive sampling to the separated feature map. Subsequently, to establish information interaction between feature layers, a channel weighting module is proposed. This module calculates weights for the separated feature map, facilitating information exchange across channels and resolutions. Moreover, it compensates for the effect of point-wise (1 ×\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\ imes$$\\end{document} 1) convolutions, filtering out more important feature information. Furthermore, a new CSPDBlock is designed, primarily composed of DSCBlock, establishing multidimensional feature correlations for each separated feature layer. This approach improves the ability to capture critical feature information and reconstruct gradient paths, thereby preserving detection accuracy. The proposed technology achieves a balance between model parameter size and detection accuracy. Furthermore, experimental results on object detection datasets demonstrate that our designed network, using fewer parameters, achieves competitive detection performance results compared to existing lightweight networks YOLOv5n, YOLOv6n, YOLOv8n, NanoDet and PP-PicoDet. The optimization effect of the designed CSPDBlock, using the VisDrone dataset, is validated when incorporated into advanced detection algorithms YOLOv5m, PPYOLOEm, YOLOv7, RTMDetm and YOLOv8m. In more detail, by incorporating the designed modules it was achieved that the parameters were reduced by 10–20% while almost maintaining detection accuracy.

Detection of cyber-attacks and network attacks using Machine Learning

The Internet and computer networks have become an important part of organizations and everyday life. New threats and challenges have emerged to wireless communication systems especially in cyber security and network attacks. The network traffic must be monitored and analysed to detect malicious activities and attacks. Recently, machine learning techniques have been applied toward the detection of network attacks. In cyber security, machine learning approaches have been utilized to handle important concerns such as intrusion detection, malware classification and detection, spam detection, and phishing detection. As a result, effective adaptive methods, such as machine learning techniques, can yield higher detection rates, lower false alarm rates and cheaper computing and transmission costs. Our key goal is detection of cyber security and network attacks such as IDS, phishing and XSS, SQL injection, respectively. The proposed strategy in this study is to employ the structure of deep neural networks for the detection phase, which should tell the system of the attack's existence in the early stages of the attack.

Research on disease diagnosis based on teacher-student network and Raman spectroscopy.

Diabetic nephropathy is a serious complication of diabetes, and primary Sjögren's syndrome is a disease that poses a major threat to women's health. Therefore, studying these two diseases is of practical significance. In the field of spectral analysis, although common Raman spectral feature selection models can effectively extract features, they have the problem of changing the characteristics of the original data. The teacher-student network combined with Raman spectroscopy can perform feature selection while retaining the original features, and transfer the performance of the complex deep neural network structure to another lightweight network structure model. This study selects five flow learning models as the teacher network, builds a neural network as the student network, uses multi-layer perceptron for classification, and selects the optimal features based on the evaluation indicators accuracy, precision, recall, and F1-score. After five-fold cross-validation, the research results show that in the diagnosis of diabetic nephropathy, the optimal accuracy rate can reach 98.3%, which is 14.02% higher than the existing research; in the diagnosis of primary Sjögren's syndrome, the optimal accuracy rate can be reached 100%, which is 10.48% higher than the existing research. This study proved the feasibility of Raman spectroscopy combined with teacher-student network in the field of disease diagnosis by producing good experimental results in the diagnosis of diabetic nephropathy and primary Sjögren's syndrome.

Effective Monoaural Speech Separation through Convolutional Top-Down Multi-View Network

Speech separation, sometimes known as the “cocktail party problem”, is the process of separating individual speech signals from an audio mixture that includes ambient noises and several speakers. The goal is to extract the target speech in this complicated sound scenario and either make it easier to understand or increase its quality so that it may be used in subsequent processing. Speech separation on overlapping audio data is important for many speech-processing tasks, including natural language processing, automatic speech recognition, and intelligent personal assistants. New speech separation algorithms are often built on a deep neural network (DNN) structure, which seeks to learn the complex relationship between the speech mixture and any specific speech source of interest. DNN-based speech separation algorithms outperform conventional statistics-based methods, although they typically need a lot of processing and/or a larger model size. This study presents a new end-to-end speech separation network called ESC-MASD-Net (effective speaker separation through convolutional multi-view attention and SuDoRM-RF network), which has relatively fewer model parameters compared with the state-of-the-art speech separation architectures. The network is partly inspired by the SuDoRM-RF++ network, which uses multiple time-resolution features with downsampling and resampling for effective speech separation. ESC-MASD-Net incorporates the multi-view attention and residual conformer modules into SuDoRM-RF++. Additionally, the U-Convolutional block in ESC-MASD-Net is refined with a conformer layer. Experiments conducted on the WHAM! dataset show that ESC-MASD-Net outperforms SuDoRM-RF++ significantly in the SI-SDRi metric. Furthermore, the use of the conformer layer has also improved the performance of ESC-MASD-Net.

BConvLSTM: a deep learning-based technique for severity prediction of a traffic crash

Predicting the severity of crashes has become a significant issue in research on road accidents. Traffic accident severity prediction is essential for protecting vulnerable road users and preventing traffic accidents. For practitioners to identify significant risk variables and set appropriate countermeasures in place, explainability of the forecast is also essential. Most previous research ignores the severity of property loss caused by traffic accidents and cannot differentiate between different levels of fatalities and property loss severity. Additionally, while an understandable structure of deep neural networks (DNN) is significantly lacking in existing works, understanding traditional systems is quite simple. An inability to use structural data when describing forecasting and the many attempts to incorporate neural networks afflict the absence of hidden layers. We propose a Deep Learning (DL) framework for forecasting traffic crash severity to overcome the accident severity prediction. It has three steps to process. Initially, collected input data are cleaned. Data cleaning is performed in a preprocessing step. We conduct experiments on two datasets, A Countrywide (US) Traffic Accident Dataset and UK Road Accident Dataset. The outcomes of the experiments demonstrate that the proposed technique outperformed other approaches and produced the best accuracy.