Fully-connected Network Research Articles

A gender specific risk assessment of coronary heart disease based on physical examination data

Large-scale screening for the risk of coronary heart disease (CHD) is crucial for its prevention and management. Physical examination data has the advantages of wide coverage, large capacity, and easy collection. Therefore, here we report a gender-specific cascading system for risk assessment of CHD based on physical examination data. The dataset consists of 39,538 CHD patients and 640,465 healthy individuals from the Luzhou Health Commission in Sichuan, China. Fifty physical examination characteristics were considered, and after feature screening, ten risk factors were identified. To facilitate large-scale CHD risk screening, a CHD risk model was developed using a fully connected network (FCN). For males, the model achieves AUCs of 0.8671 and 0.8659, respectively on the independent test set and the external validation set. For females, the AUCs of the model are 0.8991 and 0.9006, respectively on the independent test set and the external validation set. Furthermore, to enhance the convenience and flexibility of the model in clinical and real-life scenarios, we established a CHD risk scorecard base on logistic regression (LR). The results show that, for both males and females, the AUCs of the scorecard on the independent test set and the external verification set are only slightly lower (<0.05) than those of the corresponding prediction model, indicating that the scorecard construction does not result in a significant loss of information. To promote CHD personal lifestyle management, an online CHD risk assessment system has been established, which can be freely accessed at http://lin-group.cn/server/CHD/index.html.

Novel DeepONet architecture to predict stresses in elastoplastic structures with variable complex geometries and loads

A novel deep operator network (DeepONet) with a residual U-Net (ResUNet) as the trunk network is devised to predict full-field highly nonlinear elastic–plastic stress response for complex geometries obtained from topology optimization under variable loads. The proposed DeepONet uses a ResUNet in the trunk to encode complex input geometries, and a fully-connected branch network encodes the parametric loads. Additional information fusion is introduced via an element-wise multiplication of the encoded latent space to improve prediction accuracy further. The performance of the proposed DeepONet was compared to two baseline models, a standalone ResUNet and a DeepONet with fully connected networks as the branch and trunk. The results show that ResUNet and the proposed DeepONet share comparable accuracy; both can predict the stress field and accurately identify stress concentration points. However, the novel DeepONet is more memory efficient and allows greater flexibility with framework architecture modifications. The DeepONet with fully connected networks suffers from high prediction error due to its inability to effectively encode the complex, varying geometry. Once trained, all three networks can predict the full stress distribution orders of magnitude faster than finite element simulations. The proposed network can quickly guide preliminary optimization, designs, sensitivity analysis, uncertainty quantification, and many other nonlinear analyses that require extensive forward evaluations with variable geometries, loads, and other parameters. This work marks the first time a ResUNet is used as the trunk network in the DeepONet architecture and the first time that DeepONet solves problems with complex, varying input geometries under parametric loads and elasto-plastic material behavior.

High robustness single-shot wavefront sensing method using a near-field profile image and fully-connected retrieval neural network for a high power laser facility.

This paper proposes a single-shot high robustness wavefront sensing method based on deep-learning for wavefront distortion measurement in high power lasers. This method could achieve fast and robust wavefront retrieval by using a single-shot near-field profile image and trained network. The deep-learning network uses fully-skip cross connections to extract and integrate multi-scale feature maps from various layers and stages, which improves the wavefront retrieval speed and enhances the robustness of the method. The numerical simulation proves that the method could directly predict the wavefront distortion of high power lasers with high accuracy. The experiment demonstrates the residual RMS between the method and a Shack-Hartmann wavefront sensor is less than 0.01 µm. The simulational and experimental results show that the method could accurately predict the incident wavefront distortion in high power lasers, exhibiting high speed and good robustness in wavefront retrieval.

Neural network architectures for optical channel nonlinear compensation in digital subcarrier multiplexing systems.

In this work, we propose to use various artificial neural network (ANN) structures for modeling and compensation of intra- and inter-subcarrier fiber nonlinear interference in digital subcarrier multiplexing (DSCM) optical transmission systems. We perform nonlinear channel equalization by employing different ANN cores including convolutional neural networks (CNN) and long short-term memory (LSTM) layers. First, we develop a fiber nonlinearity compensation for DSCM systems based on a fully-connected network across all subcarriers. In subsequent steps, and borrowing from the perturbation analysis of fiber nonlinearity, we gradually upgrade proposed designs towards modular structures with better performance-complexity advantages. Our study shows that putting proper macro structures in design of ANN nonlinear equalizers in DSCM systems can be crucial in development of practical solutions for future generations of coherent optical transceivers.

Short-term electric load forecasting using particle swarm optimization-based convolutional neural network

Short-term electric load forecasting is essential for the operation of power systems and the power market, including economic dispatch, unit commitment, peak load shaving, load management and bidding strategy development. This paper presents a novel method that uses a hybrid convolutional neural network (CNN) that is cascaded with a fully-connected network, to conduct 24h-ahead electric load forecasting in power systems. Spearman’s rank-order correlation, which serves as input size information for the CNN, was used to measure the cross-correlation between two sets of observations (chronological loads) of time shifts or lags. Both max and average pooling operations were utilized to extract data features in concatenation while spatial and conventional dropouts were employed to avoid overfitting in the hybrid CNN. The proposed method used two loops to yield the optimal CNN. The outer loop optimizes the structure and hyperparameters (such as kernel size) of the hybrid CNN by a proposed method of enhanced elite-based particle swarm optimization (EEPSO) while the inner loop optimizes the parameters (such as values of a kernel) and weights in neural networks using the Adam optimizer. The EEPSO is based on the mean search and chaotic descending inertia weight. The proposed method was used to study the 2018 and 2019 hourly load data in Taiwan Power Company. Simulation results show that the proposed method outperforms the well-known ARIMA, Radial Basis Function Network, Support Vector Regression, Long Short-term Memory and their hybrid variants.

Neural-Network-Based Quark–Gluon Plasma Trigger for the CBM Experiment at FAIR

Algorithms optimized for high-performance computing, which ensure both speed and accuracy, are crucial for real-time data analysis in heavy-ion physics experiments. The application of neural networks and other machine learning methodologies, which are fast and have high accuracy, in physics experiments has become increasingly popular over recent years. This paper introduces a fast neural network package named ANN4FLES developed in C++, which has been optimized for use on a high-performance computing cluster for the future Compressed Baryonic Matter (CBM) experiment at the Facility for Antiproton and Ion Research (FAIR, Darmstadt, Germany). The use of neural networks for classifying events during heavy-ion collisions in the CBM experiment is under investigation. This paper provides a detailed description of the application of ANN4FLES in identifying collisions where a quark–gluon plasma (QGP) was produced. The methodology detailed here will be used in the development of a QGP trigger for event selection within the First Level Event Selection (FLES) package for the CBM experiment. Fully-connected and convolutional neural networks have been created for the identification of events containing QGP, which are simulated with the Parton–Hadron–String Dynamics (PHSD) microscopic off-shell transport approach, for central Au + Au collisions at an energy of 31.2 A GeV. The results show that the convolutional neural network outperforms the fully-connected networks and achieves over 95% accuracy on the testing dataset.

A simulation study of 1D U-Net-based inter-crystal scatter event recovery of PET detectors

To achieve high spatial resolution of reconstructed images in positron emission tomography (PET), the size of the scintillation crystal element is set small in current PET systems, which greatly increases the inter-crystal scattering (ICS) frequency. The ICS is a type of Compton scattering of the gamma photons from one crystal element to its neighborhood element, which obscures the determination of the first interaction position. In this study, we propose a 1D U-Net convolutional neural network to predict the first interaction position, which provides a universal way to efficiently solve the ICS recovery problem. The network is trained using the dataset collected from the GATE Monte Carlo simulation. The 1D U-Net structure is applied due to its capability of synthesizing both low-level and high-level information, which shows superiority in solving the ICS recovery problem. After being well trained, the 1D U-Net can generate a prediction accuracy of 78.1%. Compared to the coincidence events only composed from two photoelectric gamma photons, the sensitivity is improved by 149%. The contrast-to-noise ratio of the reconstructed contrast phantom increases from 6.973 to 10.795 for the 16 mm hot sphere. Compared to the take-energy-centroid method, the spatial resolution of the reconstructed resolution phantom can obtain the best improvement of 33.46%. Compared with the previous deep learning method based on the fully connected network, the proposed 1D U-Net can work more stably with considerably fewer network parameters. The 1D U-Net network model shows good universality when predicting different phantoms, and the computation speed is fast.

Incorporating user behavior flow for user risk assessment

PurposeInternet marketing underground industry users refer to people who use technology means to simulate a large number of real consumer behaviors to obtain marketing activities rewards illegally, which leads to increased cost of enterprises and reduced effect of marketing. Therefore, this paper aims to construct a user risk assessment model to identify potential underground industry users to protect the interests of real consumers and reduce the marketing costs of enterprises.Design/methodology/approachMethod feature extraction is based on two aspects. The first aspect is based on traditional statistical characteristics, using density-based spatial clustering of applications with noise clustering method to obtain user-dense regions. According to the total number of users in the region, the corresponding risk level of the receiving address is assigned. So that high-quality address information can be extracted. The second aspect is based on the time period during which users participate in activities, using frequent item set mining to find multiple users with similar operations within the same time period. Extract the behavior flow chart according to the user participation, so that the model can mine the deep relationship between the participating behavior and the underground industry users.FindingsBased on the real underground industry user data set, the features of the data set are extracted by the proposed method. The features are experimentally verified by different models such as random forest, fully-connected layer network, SVM and XGBOST, and the proposed method is comprehensively evaluated. Experimental results show that in the best case, our method can improve the F1-score of traditional models by 55.37%.Originality/valueThis paper investigates the relative importance of static information and dynamic behavior characteristics of users in predicting underground industry users, and whether the absence of features of these categories affects the prediction results. This investigation can go a long way in aiding further research on this subject and found the features which improved the accuracy of predicting underground industry users.

Two-Stream Graph Convolutional Network-Incorporated Latent Feature Analysis

Historical Quality-of-Service (QoS) data describing existing user-service invocations are vital to understanding user behaviors and cloud service conditions. Collaborative Filtering (CF) models based on Matrix Factorization (MF) have proven to be highly efficient in performing representation learning in QoS data. However, its performance is hindered by its linear inherence and implicit encoding of collaborative QoS signal. To address this critical issue, we present a novel approach in this paper, dubbed as <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <u>T</u></b> wo-stream <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <u>G</u></b> raph convolutional network-incorporated <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <u>L</u></b> atent <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <u>F</u></b> eature <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <u>A</u></b> nalysis (TGLFA). The proposed TGLFA is significantly different from previous approaches to representation learning in QoS data in the following three aspects. First, it constructs a multilayer fully-connected network to capture the attribute characteristics representing the nonlinear latent features of service. Second, TGLFA constructs a biparty graph to represent the user-service interactions, where the light graph convolutional network is adopted to acquire the high-order connectivity in QoS data. Last, Aiming to improve computational efficiency, the proposed approach considers the mechanism for data density-oriented modeling when building the input and output layers. Detailed experimental results on eight large-scale cases constructed on two real QoS datasets demonstrate that the proposed TGLFA significantly outperforms its state-of-the-art peers in both estimation accuracy for missing QoS data and computational efficiency. The notable results show that TGLFA is a novel and effective approach to QoS data representation learning.

Speech Emotion Recognition Based on Deep Residual Shrinkage Network

Speech emotion recognition (SER) technology is significant for human–computer interaction, and this paper studies the features and modeling of SER. Mel-spectrogram is introduced and utilized as the feature of speech, and the theory and extraction process of mel-spectrogram are presented in detail. A deep residual shrinkage network with bi-directional gated recurrent unit (DRSN-BiGRU) is proposed in this paper, which is composed of convolution network, residual shrinkage network, bi-directional recurrent unit, and fully-connected network. Through the self-attention mechanism, DRSN-BiGRU can automatically ignore noisy information and improve the ability to learn effective features. Network optimization, verification experiment is carried out in three emotional datasets (CASIA, IEMOCAP, and MELD), and the accuracy of DRSN-BiGRU are 86.03%, 86.07%, and 70.57%, respectively. The results are also analyzed and compared with DCNN-LSTM, CNN-BiLSTM, and DRN-BiGRU, which verified the superior performance of DRSN-BiGRU.

Hyperspectral Remote Sensing Images Feature Extraction Based on Spectral Fractional Differentiation

To extract effective features for the terrain classification of hyperspectral remote-sensing images (HRSIs), a spectral fractional-differentiation (SFD) feature of HRSIs is presented, and a criterion for selecting the fractional-differentiation order is also proposed based on maximizing data separability. The minimum distance (MD) classifier, support vector machine (SVM) classifier, K-nearest neighbor (K-NN) classifier, and logistic regression (LR) classifier are used to verify the effectiveness of the proposed SFD feature, respectively. The obtained SFD feature is sent to the full connected network (FCN) and 1-dimensionality convolutional neural network (1DCNN) for deep-feature extraction and classification, and the SFD-Spa feature cube containing spatial information is sent to the 3-dimensionality convolutional neural network (3DCNN) for deep-feature extraction and classification. The SFD-Spa feature after performing the principal component analysis (PCA) on spectral pixels is directly connected with the first principal component of the original data and sent to 3DCNNPCA and hybrid spectral net (HybridSN) models to extract deep features. Experiments on four real HRSIs using four traditional classifiers and five network models have shown that the extracted SFD feature can effectively improve the accuracy of terrain classification, and sending SFD feature to deep-learning environments can further improve the accuracy of terrain classification for HRSIs, especially in the case of small-size training samples.

Convolutional autoencoder based on latent subspace projection for anomaly detection.

Image anomaly detection (AD) is widely researched on many occasions in computer vision tasks. High-dimensional data, such as image data, with noise and complex background is still challenging to detect anomalies under the situation that imbalanced or incomplete data are available. Some deep learning methods can be trained in an unsupervised way and map the original input into low-dimensional manifolds to predict larger differences in anomalies according to normal ones by dimension reduction. However, training a single low-dimension latent space is limited to present the low-dimensional features due to the fact that the noise and irreverent features are mapped into this space, resulting in that the manifolds are not discriminative for detecting anomalies. To address this problem, a new autoencoder framework is proposed in this study with two trainable mutually orthogonal complementary subspaces in the latent space, by latent subspace projection (LSP) mechanism, which is named as LSP-CAE. Specifically, latent subspace projection is used to train the latent image subspace (LIS) and the latent kernel subspace (LKS) in the latent space of the autoencoder-like model respectively, which can enhance learning power of different features from the input instance. The features of normal data are projected into the latent image subspace, while the latent kernel subspace is trained to extract the irrelevant information according to normal features by end-to-end training. To verify the generality and effectiveness of the proposed method, we replace the convolutional network with the fully-connected network contucted in the real-world medical datasets. The anomaly score based on projection norms in two subspace is used to evaluate the anomalies in the testing. Consequently, our proposed method can achieve the best performance according to four public datasets in comparison of the state-of-the-art methods.

A fully-connected three-user quantum hyperentangled network

Exploiting the fantastic features of quantum mechanics, a hyperentangled quantum network encoded in multiple degree of freedoms (DOF), e.g., polarization and orbital angular momentum DOFs, can encode more qubits per transmitted photon and offers a promising platform for many dramatic applications. Here, we demonstrate such a hyperentangled multiuser network with a fully connected network architecture by using dense wavelength division multiplexing and entanglement transfer technique. Three hyperentangled states in polarization and time-energy DOFs are multiplexed to three single mode fibers to form the fully connected network architecture. Then, three interferometric quantum gates are utilized for transferring quantum entanglement from time-energy to orbital angular momentum DOF. The experimental results reveal a high quality of the hyperentanglement of the constructed network with the entangled state fidelity of higher than 96%. Our approach can provide a novel way to construct a large-scale hyperentangled network that can support various kinds of quantum tasks like superdense coding and teleportation.

Motor decoding from the posterior parietal cortex using deep neural networks

Objective. Motor decoding is crucial to translate the neural activity for brain-computer interfaces (BCIs) and provides information on how motor states are encoded in the brain. Deep neural networks (DNNs) are emerging as promising neural decoders. Nevertheless, it is still unclear how different DNNs perform in different motor decoding problems and scenarios, and which network could be a good candidate for invasive BCIs. Approach. Fully-connected, convolutional, and recurrent neural networks (FCNNs, CNNs, RNNs) were designed and applied to decode motor states from neurons recorded from V6A area in the posterior parietal cortex (PPC) of macaques. Three motor tasks were considered, involving reaching and reach-to-grasping (the latter under two illumination conditions). DNNs decoded nine reaching endpoints in 3D space or five grip types using a sliding window approach within the trial course. To evaluate decoders simulating a broad variety of scenarios, the performance was also analyzed while artificially reducing the number of recorded neurons and trials, and while performing transfer learning from one task to another. Finally, the accuracy time course was used to analyze V6A motor encoding. Main results. DNNs outperformed a classic Naïve Bayes classifier, and CNNs additionally outperformed XGBoost and Support Vector Machine classifiers across the motor decoding problems. CNNs resulted the top-performing DNNs when using less neurons and trials, and task-to-task transfer learning improved performance especially in the low data regime. Lastly, V6A neurons encoded reaching and reach-to-grasping properties even from action planning, with the encoding of grip properties occurring later, closer to movement execution, and appearing weaker in darkness. Significance. Results suggest that CNNs are effective candidates to realize neural decoders for invasive BCIs in humans from PPC recordings also reducing BCI calibration times (transfer learning), and that a CNN-based data-driven analysis may provide insights about the encoding properties and the functional roles of brain regions.

Multi-source partial discharge diagnosis in gas-insulated switchgear via zero-shot learning

Partial discharge (PD) diagnosis in gas insulation switchgear (GIS) has been continuously studied as a hot issue. The diagnostic model developed at this stage requires the possible combination of all single PDs, and each combination has sufficient data. However, the demand for the number of multi-source PD samples increases in the form of an index due to the accidental and concurrency of different types of PD, where it is unrealistic to collect sufficient PD samples. To this end, we proposed zero-shot learning (ZL) to diagnose multi-source PD. The ZL diagnosis multi-source PD only required single PD for training. Specifically, an attention residual network was introduced for feature extraction. Then, a novel semantic descriptor was proposed to obtain the semantic attributes, and the semantic embedded features were mapped to the visual space with a full connection network. Finally, the relationship network adopted the similarity between the visual features and semantic embedded features, and the multi-source PD diagnosis was realized. In addition, the multi-source PD experiment in GIS was constructed to verify the proposed ZL. The experimental results demonstrated that the ZL could achieve 90.25 % accuracy when only single PD participated in the training. It provided novel solutions for the diagnosis of multi-source PD in GIS.

A feasibility study of multi-electrode high-purity germanium detector for 76Ge neutrinoless double beta decay searching

Experiments to search for neutrinoless double-beta (0νββ) decay of 76Ge using a high-purity germanium (HPGe) detector rely heavily on background suppression technologies to enhance their sensitivities. In this work, we proposed a pulse-shape analysis method based on a neural network (NN) and a light gradient boosting machine (lightGBM; LGB) to discriminate single-electron (background) and double-electrons (0νββ signal) events in a multi-electrode HPGe detector. In this paper, we describe a multi-electrode HPGe detector system, a data-processing system, and pulse-shape simulation procedures. We built a fully connected (FC) neural network and an LGB model to classify the single- and double-electron events. The FC network is trained with simulated single- and double-electron-induced pulses and tested in an independent dataset generated by the pulse-shape simulation. The discrimination efficiency of the FC neural network in the test set for the 0νββ double-electron events signal was 77.4%, the precision was 57.7%, and the training time was 430 min. The discrimination efficiency of LGB model was 73.1%, the precision was 64.0%, and the training time was 1.5 min. This study demonstrated that it is feasible to realize single- and double-electron discrimination on multi-electrode HPGe detectors using an FC neural network and LGB model. These results can be used as a reference for future 76Ge 0νββ experiments.

Hierarchical-Absolute Reciprocity Calibration for Millimeter-Wave Hybrid Beamforming Systems

In time-division duplexing (TDD) millimeter-wave (mmWave) massive multiple-input multiple-output (MIMO) systems, the reciprocity mismatch severely degrades the performance of the hybrid beamforming (HBF). In this work, to mitigate the detrimental effect of the reciprocity mismatch, we investigate reciprocity calibration for the mmWave-HBF system with a fully-connected phase shifter network. To reduce the over-head and computational complexity of reciprocity calibration, we first decouple digital radio frequency (RF) chains and analog RF chains with beamforming design. Then, the entire calibration problem of the HBF system is equivalently decomposed into two subproblems corresponding to the digital-chain calibration and analog-chain calibration. To solve the calibration problems efficiently, a closed-form solution to the digital-chain calibration problem is derived, while an iterative-alternating optimization algorithm for the analog-chain calibration problem is proposed. To measure the performance of the proposed algorithm, we derive the Cramér-Rao lower bound on the errors in estimating mismatch coefficients. The results reveal that the estimation errors of mismatch coefficients of digital and analog chains are uncorrelated, and that the mismatch coefficients of receive digital chains can be estimated perfectly. Simulation results are presented to validate the analytical results and to show the performance of the proposed calibration approach.

Digital Hardware Implementations of Spiking Neural Networks With Selective Input Sparsity for Edge Inferences in Controlled Image Acquisition Environments

Digital hardware implementations of Spiking Neural Networks designed using Selective Input Sparsity (SIS) are proposed for edge inference applications in image classification where the image acquisition environment is controlled. These sparsely connected networks are well-suited to area-constrained applications as they require fewer neurons and synapses than baseline Fully Connected (FC) networks of analogous structures. The SIS networks were validated on FPGA, and baseline FC networks were also implemented on FGPA for comparison. The SIS networks require fewer hardware resources and make inferences faster than the baseline FC networks without substantial impact on the classification accuracy.

Inception-embedded attention memory fully-connected network for short-term wind power prediction

With the increasing demand for energy in the world today, wind energy has turned out to be an attractive alternative to traditional fossil energy sources because of the characteristics of being clean, non-polluting, and easily accessible. Reliably predicting wind power is vital to improving energy utilization and ensuring the stability of power system operation. However, because of the uncertainty and instability of wind energy, accurately predicting wind power is still challenging. Therefore, this study proposes an Inception-embedded attention memory fully-connected network short-term wind power prediction model, incorporating improved attention mechanisms. As a result, the Inception-embedded attention memory fully-connected network can give reliable wind power predictions. This study utilizes a dataset of about 400 days from Natal and compares the Inception-embedded attention memory fully-connected network with 23 algorithms including EffiientNet, NasNet, and ResNet. The comparison results show that the Inception-embedded attention memory fully-connected network obtains reliable wind power prediction one day ahead and outperforms all other compared algorithms by more than 40% in all evaluation metrics.

Convolutional Deep Neural Network and Full Connectivity for Speech Enhancement

The speech signal that is received in real-time has background noise and reverberations, which have an impact on the quality of speech. Therefore, it is crucial to reduce or eliminate the noise and increase the intelligibility and quality of speech signals. In this study, a proposed method that is the most effective and challenging in a low SNR environment for three types of noise are removed, including washing machine, traffic noise, and electric fan noise, and clean speech is recovered. with three samples of noise which are mixed and added to the clean speech signal with a lower level of SNR value fixed at (-5, 0, 5) dBs, that noise source takes equal weights. The enhancement of the corrupted speech signal is done by applying a fully connected and convolutional neural network-based denoising algorithm and comparing their performance. The proposed network shows that a fully connected network (FCN) has less elapsed time than a convolutional network (CNN) while still achieving better performance, demonstrating its applicability for an embedded system. Also, the results obtained show that, overall, the CNN is better than the FCN regarding maximum coloration, PSNR, MES, and STOI.