Conventional Deep Neural Networks Research Articles

Top-Heavy CapsNets Based on Spatiotemporal Non-Local for Action Recognition

To effectively comprehend human actions, we have developed a Deep Neural Network (DNN) that utilizes inner spatiotemporal non-locality to capture meaningful semantic context for efficient action identification. This work introduces the Top-Heavy CapsNet as a novel approach for video analysis, incorporating a 3D Convolutional Neural Network (3DCNN) to apply the thematic actions of local classifiers for effective classification based on motion from the spatiotemporal context in videos. This DNN comprises multiple layers, including 3D Convolutional Neural Network (3DCNN), Spatial Depth-Based Non-Local (SBN) layer, and Deep Capsule (DCapsNet). Firstly, the 3DCNN extracts structured and semantic information from RGB and optical flow streams. Secondly, the SBN layer processes feature blocks with spatial depth to emphasize visually advantageous cues, potentially aiding in action differentiation. Finally, DCapsNet is more effective in exploiting vectorized prominent features to represent objects and various action features for the ultimate label determination. Experimental results demonstrate that the proposed DNN achieves an average accuracy of 97.6%, surpassing conventional DNNs on the traffic police dataset. Furthermore, the proposed DNN attains average accuracies of 98.3% and 80.7% on the UCF101 and HMDB51 datasets, respectively. This underscores the applicability of the proposed DNN for effectively recognizing diverse actions performed by subjects in videos.

A Biologically Inspired Movement Recognition System with Spiking Neural Networks for Ambient Assisted Living Applications.

This study presents a novel solution for ambient assisted living (AAL) applications that utilizes spiking neural networks (SNNs) and reconfigurable neuromorphic processors. As demographic shifts result in an increased need for eldercare, due to a large elderly population that favors independence, there is a pressing need for efficient solutions. Traditional deep neural networks (DNNs) are typically energy-intensive and computationally demanding. In contrast, this study turns to SNNs, which are more energy-efficient and mimic biological neural processes, offering a viable alternative to DNNs. We propose asynchronous cellular automaton-based neurons (ACANs), which stand out for their hardware-efficient design and ability to reproduce complex neural behaviors. By utilizing the remote supervised method (ReSuMe), this study improves spike train learning efficiency in SNNs. We apply this to movement recognition in an elderly population, using motion capture data. Our results highlight a high classification accuracy of 83.4%, demonstrating the approach's efficacy in precise movement activity classification. This method's significant advantage lies in its potential for real-time, energy-efficient processing in AAL environments. Our findings not only demonstrate SNNs' superiority over conventional DNNs in computational efficiency but also pave the way for practical neuromorphic computing applications in eldercare.

Wind Shear and Aircraft Aborted Landings: A Deep Learning Perspective for Prediction and Analysis

In civil aviation, severe weather conditions such as strong wind shear, crosswinds, and thunderstorms near airport runways often compel pilots to abort landings to ensure flight safety. While aborted landings due to wind shear are not common, they occur under specific environmental and situational circumstances. This research aims to accurately predict aircraft aborted landings using three advanced deep learning techniques: the conventional deep neural network (DNN), the deep and cross network (DCN), and the wide and deep network (WDN). These models are supplemented by various data augmentation methods, including the Synthetic Minority Over-Sampling Technique (SMOTE), KMeans-SMOTE, and Borderline-SMOTE, to correct the imbalance in pilot report data. Bayesian optimization was utilized to fine-tune the models for optimal predictive accuracy. The effectiveness of these models was assessed through metrics including sensitivity, precision, F1-score, and the Matthew Correlation Coefficient. The Shapley Additive Explanations (SHAP) algorithm was then applied to the most effective models to interpret their results and identify key factors, revealing that the intensity of wind shear, specific runways like 07R, and the vertical distance of wind shear from the runway (within 700 feet above runway level) were significant factors. The results of this research provide valuable insights to civil aviation experts, potentially revolutionizing safety protocols for managing aborted landings under adverse weather conditions, thereby improving overall airport efficiency and safety.

Speckle noise reduction on aligned consecutive ultrasound frames via deep neural network

Despite the benefits of ultrasound (US) imaging systems for medical diagnosis and treatment, US images are prone to low resolution and contrast due to US’s inherent attributes, as well as affected by speckle noise that directly influences their quality. In retrospective studies, diverse filters have been applied to minimize the effects of speckle noise and enhance the quality of US images. In this article, we propose a method of enhancing US images inspired by synthetic aperture imaging, which provides high-resolution images by adding low-resolution images and measuring the probe’s movement. Our proposed method does not involve synthetic aperture imaging but compensates for the motion effect in the temporal dimension, aligns consecutive images, and stacks aligned images to suppress speckle noise and consequently enhance the resolution of US images. We exploited deep neural network (DNN) models to estimate motion parameters between consecutive US images. In a new database of US images, we also collected the images’ position-related information implicitly measured in inertial measurement units, which was exploited as a ground truth for motion parameters between consecutive images. Compared with other image-enhancing techniques involving conventional filters and DNN modalities, our method demonstrated superiority in enhancing the quality of US images. We also found that estimating motion parameters directly influenced the success of the image-stacking process. As in ablation studies in DNNs, we additionally investigated the effect of dropping some images in the temporal dimension, which revealed that contextual differences and excessive rates of movement in successive US images weakens the image-stacking process and thus the potential enhancement of US images.

K-Means-Based DNN Algorithm for a High Accuracy VLP System

In this paper, a positioning algorithm based on the combination of K-means clustering and deep neural networks (DNNs) is first presented for multiple light emitting diodes (LEDs) integrated with visible light positioning (VLP) systems. We extracted the maximum value from the collected optical power of LEDs, utilizing the ratio of each optical power to this maximum optical power as the input training data. The experimental results demonstrate that the proposed algorithm outperformed the conventional DNN algorithm in terms of anti-jamming capability and positioning accuracy. In addition, the positioning accuracy of the proposed system reached a millimeter level, which is the highest experimental VLP accuracy, to the best of our knowledge.

Rapid screening of chemicals for their potential to cause specific toxidromes

Toxidromes constitute patterns of symptoms and signs caused by specific toxic effects that guide emergency treatments. Computational identification of chemicals that cause different toxidromes allows us to rapidly screen novel compounds and compound classes as to their potential toxicity. The aim of the current study was to create a computational toolset that can map chemicals to their potential toxidromes. Hence, we evaluated the performance of a state-of-the-art deep learning method—the recently developed communicative message passing neural network (CMPNN)—for its ability to overcome the use of small datasets for training deep learning models. Our results indicated that multi-task training—a technique known for its ability to use multiple small datasets to train conventional deep neural networks—works equally well with CMPNN. We also showed that CMPNN-based ensemble learning results in more reliable predictions than those obtained using a single CMPNN model. In addition, we showed that the standard deviations of individual model predictions from an ensemble of CMPNN models correlated with the errors of ensemble predictions and could be used to estimate the reliability of ensemble predictions. For toxidromes that do not have well-defined molecular mechanisms or sufficient data to train a deep learning model, we used the similarity ensemble approach to develop molecular structural similarity-based toxidrome models. We made the toolset developed in this study publicly accessible via a web user interface at https://toxidrome.bhsai.org/.

Collective Decision of One-vs-Rest Networks for Open-Set Recognition.

Unknown examples that are unseen during training often appear in real-world pattern recognition tasks, and an intelligent self-learning system should be able to distinguish between known examples and unknown examples. Accordingly, open-set recognition (OSR), which addresses the problem of classifying knowns and identifying unknowns, has recently been highlighted. However, conventional deep neural networks (DNNs) using a softmax layer are vulnerable to overgeneralization, producing high confidence scores for unknowns. In this article, we propose a simple OSR method that is based on the intuition that the OSR performance can be maximized by setting strict and sophisticated decision boundaries that reject unknowns while maintaining satisfactory classification performance for knowns. For this purpose, a novel network structure, in which multiple one-vs-rest networks (OVRNs) follow a convolutional neural network (CNN) feature extractor, is proposed. Here, an OVRN is a simple feedforward neural network that is designed to assign confidence scores that are lower than those in the softmax layer to unknown samples so that unknown samples can be more effectively separated from known classes. Furthermore, the collective decision score is modeled by combining the multiple decisions reached by the OVRNs to alleviate overgeneralization. Extensive experiments were conducted on various datasets, and the experimental results show that the proposed method performs significantly better than the state-of-the-art methods by effectively reducing overgeneralization. The code is available at https://github.com/JaeyeonJang/Openset-collective-decision.

ScHybridBERT: integrating gene regulation and cell graph for spatiotemporal dynamics in single-cell clustering.

Graph learning models have received increasing attention in the computational analysis of single-cell RNA sequencing (scRNA-seq) data. Compared with conventional deep neural networks, graph neural networks and language models have exhibited superior performance by extracting graph-structured data from raw gene count matrices. Established deep neural network-based clustering approaches generally focus on temporal expression patterns while ignoring inherent interactions at gene-level as well as cell-level, which could be regarded as spatial dynamics in single-cell data. Both gene-gene and cell-cell interactions are able to boost the performance of cell type detection, under the framework of multi-view modeling. In this study, spatiotemporal embedding and cell graphs are extracted to capture spatial dynamics at the molecular level. In order to enhance the accuracy of cell type detection, this study proposes the scHybridBERT architecture to conduct multi-view modeling of scRNA-seq data using extracted spatiotemporal patterns. In this scHybridBERT method, graph learning models are employed to deal with cell graphs and the Performer model employs spatiotemporal embeddings. Experimental outcomes about benchmark scRNA-seq datasets indicate that the proposed scHybridBERT method is able to enhance the accuracy of single-cell clustering tasks by integrating spatiotemporal embeddings and cell graphs.

Go Deep or Broad? Exploit Hybrid Network Architecture for Weakly Supervised Object Classification and Localization.

Weakly supervised object classification and localization are learned object classes and locations using only image-level labels, as opposed to bounding box annotations. Conventional deep convolutional neural network (CNN)-based methods activate the most discriminate part of an object in feature maps and then attempt to expand feature activation to the whole object, which leads to deteriorating the classification performance. In addition, those methods only use the most semantic information in the last feature map, while ignoring the role of shallow features. So, it remains a challenge to enhance classification and localization performance with a single frame. In this article, we propose a novel hybrid network, namely deep and broad hybrid network (DB-HybridNet), which combines deep CNNs with a broad learning network to learn discriminative and complementary features from different layers, and then integrates multilevel features (i.e., high-level semantic features and low-level edge features) in a global feature augmentation module. Importantly, we exploit different combinations of deep features and broad learning layers in DB-HybridNet and design an iterative training algorithm based on gradient descent to ensure the hybrid network work in an end-to-end framework. Through extensive experiments on caltech-UCSD birds (CUB)-200 and imagenet large scale visual recognition challenge (ILSVRC) 2016 datasets, we achieve state-of-the-art classification and localization performance.

Intelligent reconstruction for spatially irregular seismic data by combining compressed sensing with deep learning

Data reconstruction is the most essential step in seismic data processing. Although the compressed sensing (CS) theory breaks through the Nyquist sampling theorem, we previously proved that the CS-based reconstruction of spatially irregular seismic data could not fully meet the theoretical requirements, resulting in low reconstruction accuracy. Although deep learning (DL) has great potential in mining features from data and accelerating the process, it faces challenges in earth science such as limited labels and poor generalizability. To improve the generalizability of deep neural network (DNN) in reconstructing seismic data in the actual situation of limited labeling, this paper proposes a method called CSDNN that combines model-driven CS and data-driven DNN to reconstruct the spatially irregular seismic data. By physically constraining neural networks, this method increases the generalizability of the network and improves the insufficient reconstruction caused by the inability to sample randomly in the whole data definition domain. Experiments on the synthetic and field seismic data show that the CSDNN reconstruction method achieves better performance compared with the conventional CS method and DNN method, including those with low sampling rates, which verifies the feasibility, effectiveness and generalizability of this approach.

Rolling bearing fault diagnosis method using time-frequency information integration and multi-scale TransFusion network

Advances in deep learning methods have demonstrated remarkable development in diagnosing faults of rotating machinery. The currently popular deep neural networks suffer from design flaws in their network structure, leading to issues of long-term dependencies in fault diagnosis models built upon conventional deep neural networks. Consequently, such models exhibit insufficient global perceptual capabilities towards fault features. Furthermore, how accurately pre-trained models can diagnose faults is hugely impacted by changes in bearings’ working conditions. To tackle the aforementioned issues, this study puts forth a multi-scale TransFusion (MSTF) model for diagnosing faults in rolling bearings under multiple operating conditions. Firstly, a time-frequency symmetric dot pattern transformation technique is designed to transform the original vibration signals into two-dimensional representations. This method can effectively highlight the distinctions between different fault types. Secondly, a multi-scale feature fusion module is established, which fully extracts low-level features from the time-frequency signals and reduces the complexity of the subsequent attention calculations. Meanwhile, relying on the advantages of the Transformer model in capturing global dependencies, the long-range periodic fault information is deeply mined. Finally, multi-head and multi-layer attention are visualized to enhance the interpretability of the model. After analyzing two case studies with both public and experimental datasets, the examination demonstrated that the developed model outperformed other state-of-the-art models. The diagnostic model developed in this study exhibits the ability to accurately diagnose bearing faults across multiple operating conditions while maintaining high robustness to signals contaminated with noise.

MLRNet: A Meta-Loss Reweighting Network for Biased Data on Text Classification

Artificially generated datasets often exhibit biases, leading conventional deep neural networks to overfit. Typically, a weighted function adjusts sample impact during model updates using weighted loss. Meta-neural networks, trained with meta-learning principles, generalize well across tasks, acquiring generalized weights. This enables the self-generation of tailored weighted functions for data biases. However, datasets may simultaneously exhibit imbalanced classes and corrupted labels, posing a challenge for current meta-models. To address this, this paper presents Meta-Loss Reweighting Network (MLRNet) with fusion attention features. MLRNet continually evolves sample loss values, integrating them with sample features from self-attention layers in a semantic space. This enhances discriminative power for biased samples. By employing minimal unbiased meta-data for guidance, mutual optimization between the classifier and the meta-model is conducted, endowing biased samples with more reasonable weights. Experiments on English and Chinese benchmark datasets including artificial and real-world biased data show MLRNet’s superior performance under biased data conditions.

Supervised Contrastive Learning for Fault Diagnosis Based on Phase-Resolved Partial Discharge in Gas-Insulated Switchgear

Supervised contrastive learning (SCL) has recently emerged as an alternative to conventional machine learning and deep neural networks. In this study, we propose an SCL model with data augmentation techniques using phase-resolved partial discharge (PRPD) in gas-insulated switchgear (GIS). To increase the fault data for training, we employ Gaussian noise adding, Gaussian noise scaling, random cropping, and phase shifting for supervised contrastive loss. The performance of the proposed SCL was verified by four types of faults in the GIS and on-site noise using an on-line ultra-high-frequency (UHF) partial discharge (PD) monitoring system. The experimental results show that the proposed SCL achieves a classification accuracy of 97.28% and outperforms the other algorithms, including support vector machines (SVM), multilayer perceptron (MLP), and convolution neural networks (CNNs) in terms of classification accuracy, by 6.8%, 4.28%, 2.04%, respectively.

Transfer learning approach to reduce similar IOT sensor data for industrial applications

The Industrial Internet of Things (IIoT) is a developing technology that has the potential to advance industrial intelligence, achieve targeted effectiveness, and lower manufacturing costs. Data fusion, a procedure that enables the collection, analysis, and processing of the enormous amounts of Internet of Things (IoT) information created by industrialized machinery and applications, is essential to IIoT's advancement and improvement of industrial products and applications. A real-time, efficient, and private information image fusion mechanism is required by the IIoT. Moreover, the current efforts call for such training of multiple systems in data analysis that also fall short of meeting real-time IIoT needs. The ineffectiveness of internal defenses and the challenge of finding the right balance between system stability and data confidentiality, however, undermine the efficiency and privacy preservation of a data fusion process. In this paper, we propose a Transfer learning-based Secure Data Fusion (TSDF) for the IIoT to address the aforementioned issues. In addition to the proposed network, we developed and deployed a defect estimation method that makes use of sensor data fusion to evaluate the manufacturing plant's operational state. The efficiency outcomes of evaluating the conceptual model are included in the original study, including a comparison to a conventional deep neural network structure. The study's findings demonstrate that TSDF may provide privacy, preserving information fusion in diverse IIoT application environments while reaching higher system capacity and low latency.

ULEEN: A Novel Architecture for Ultra-low-energy Edge Neural Networks

‘‘Extreme edge” 1 devices, such as smart sensors, are a uniquely challenging environment for the deployment of machine learning. The tiny energy budgets of these devices lie beyond what is feasible for conventional deep neural networks, particularly in high-throughput scenarios, requiring us to rethink how we approach edge inference. In this work, we propose ULEEN, a model and FPGA-based accelerator architecture based on weightless neural networks (WNNs). WNNs eliminate energy-intensive arithmetic operations, instead using table lookups to perform computation, which makes them theoretically well-suited for edge inference. However, WNNs have historically suffered from poor accuracy and excessive memory usage. ULEEN incorporates algorithmic improvements and a novel training strategy inspired by binary neural networks (BNNs) to make significant strides in addressing these issues. We compare ULEEN against BNNs in software and hardware using the four MLPerf Tiny datasets and MNIST. Our FPGA implementations of ULEEN accomplish classification at 4.0–14.3 million inferences per second, improving area-normalized throughput by an average of 3.6× and steady-state energy efficiency by an average of 7.1× compared to the FPGA-based Xilinx FINN BNN inference platform. While ULEEN is not a universally applicable machine learning model, we demonstrate that it can be an excellent choice for certain applications in energy- and latency-critical edge environments.

Psychophysics may be the game-changer for deep neural networks (DNNs) to imitate the human vision.

Psychologically faithful deep neural networks (DNNs) could be constructed by training with psychophysics data. Moreover, conventional DNNs are mostly monocular vision based, whereas the human brain relies mainly on binocular vision. DNNs developed as smaller vision agent networks associated with fundamental and less intelligent visual activities, can be combined to simulate more intelligent visual activities done by the biological brain.

Adaptive deep neural networks for solving corner singular problems

Deep neural networks (DNNs) for numerical solutions to partial differential equations (PDEs) have exhibited their remarkable merits of meshless methods, dimensionless features, and nonlinear approximation powers. The majority of conventional DNNs are mainly concentrated on the low- or high-dimensional PDE with smooth solutions. In singular problems the DNN does not behave so efficiently as in the smooth problem because of low regularities of the solution. In this paper, we propose a novel adaptive DNN for the PDE with corner singular solutions. In the corner singular problem gradients of solution vary rapidly around some particular vertices or lines. This poses a difficulty in development of DNN solvers. We design adaptive DNN techniques for the corner singularity, which incorporates (a) an adaptive loss function that assigns radial weights to local losses, (b) an adaptive activation function that varies the choice of activation functions from neuron to neuron, and (c) an adaptive sampling strategy that selectively focuses on sampling points near the singular locations. The adaptive DNN with these adaptive schemes greatly enhances approximation accuracy and computation efficiency. This is verified by a comprehensive series of numerical experiments of 2D and 3D corner singular problems. The comparison analyzes with the existing DNN solvers are also made to demonstrate effectiveness of the new method. The L2 relative errors of the proposed adaptive DNN are reduced by an order of magnitude with respect to several conventional DNN methods.

REKP: Refined External Knowledge into Prompt-Tuning for Few-Shot Text Classification

Text classification is a machine learning technique employed to assign a given text to predefined categories, facilitating the automatic analysis and processing of textual data. However, an important problem is that the number of new text categories is growing faster than that of human annotation data, which makes many new categories of text data lack a lot of annotation data. As a result, the conventional deep neural network is forced to over-fit, which damages the application in the real world. As a solution to this problem, academics recommend addressing data scarcity through few-shot learning. One of the efficient methods is prompt-tuning, which transforms the input text into a mask prediction problem featuring [MASK]. By utilizing descriptors, the model maps output words to labels, enabling accurate prediction. Nevertheless, the previous prompt-based adaption approaches often relied on manually produced verbalizers or a single label to represent the entire label vocabulary, which makes the mapping granularity low, resulting in words not being accurately mapped to their label. To address these issues, we propose to enhance the verbalizer and construct the refined external knowledge into a prompt-tuning (REKP) model. We employ the external knowledge bases to increase the mapping space of tagged terms and design three refinement methods to remove noise data. We conduct comprehensive experiments on four benchmark datasets, namely AG’s News, Yahoo, IMDB, and Amazon. The results demonstrate that REKP can outperform the state-of-the-art baselines in terms of Micro-F1 on knowledge-enhanced text classification. In addition, we conduct an ablation study to ascertain the functionality of each module in our model, revealing that the refinement module significantly contributes to enhancing classification accuracy.

A high-precision retrieval method for methane vertical profiles based on dual-band spectral data from the GOSAT satellite

Satellite remote sensing has become an important method of monitoring methane concentrations. The full-physics algorithm is widely used for methane vertical profile retrieval, but the method is relatively time-consuming. Most current studies have focused on the retrieval of methane vertical profiles from thermal infrared (TIR) spectral data only, resulting in low accuracy in the near-surface retrieval results. To obtain the atmospheric methane vertical profile from satellite sounding data quickly and accurately, this paper proposes a high-precision retrieval method based on the ResNet18 model, a conventional and effective deep convolutional neural network, using TIR band and shortwave infrared (SWIR) band spectral data from the GOSAT. The model performs well on the test set, with mean absolute error (MAE) below 12.78 ppb and root mean square error (RMSE) below 23.71 ppb with pressure greater than 150 hPa. The accuracy of the retrieval results and GOSAT CH4 vertical profile products are validated using airborne measurement data. In the mid-latitude region, using NOAA airborne measurement data as reference values, the average of MAE and RMSE of the retrieval results between 500 hPa and 900 hPa are reduced by 12.05 ppb and 14.07 ppb, respectively, compared to the GOSAT CH₄ vertical profile product. In the low-latitude region, using CARIBIC airborne measurement data as reference values, the average of MAE and RMSE between 300 hPa and 600 hPa are reduced by 11.18 ppb and 12.20 ppb, respectively. The validation results show that the methane vertical profile can be quickly and accurately retrieved based on the ResNet18 model, combined with dual-band spectral data, temperature, and surface pressure.

Enhanced capsule network‐based executable files malware detection and classification—deep learning approach

SummaryMalware has considerably increased recently, posing a serious security danger to both people and enterprises. In order to distinguish and stop the negative effects of malware, a variety of machine and deep learning approaches have been used to detect it. However, while extracting malware features, the feature‐to‐feature spatial hierarchy is not taken into account by the existing techniques and as a result, information is lost during the pooling operation. Hence, a modified capsule deep neural network was developed in which discriminative features are extracted from three channel image derived from malware binary with considering feature‐to‐feature spatial hierarchy. Also, conventional capsule deep neural network is modified by adding a global average pooling layer before fully connected layer thereby classified the dataset as malicious or benign without any loss of information. Moreover, these malwares were not accurately classified based on their families using existing variants of convolutional neural network (CNN) since malware family variants can modify due to minute changes in malware binaries. Hence, a hybrid deep convolutional neural network (DCNN) and long‐short‐term memory (LSTM) has been utilized that determine minute changes in malware binaries using LSTM without vanishing gradient issue and effectively perform malware family classification using DCNN. As a result, the proposed approach successfully identifies malware in executable files and categorizes malware into families with 98.5% accuracy.