Neural Layers Research Articles

KAZAKH SIGN LANGUAGE RECOGNITION BY USING MACHINE LEARNING METHODS

Pattern recognition involves identifying objects within images, where the objects are abstract and can vary in shape. We have chosen to explore the relevant area of gesture recognition. To recognize Kazakh Sign Language (KSL), it is essential first to learn the Kazakh Sign Alphabet (KSA). Training a neural network to recognize KSL requires collecting datasets in the form of images depicting hand gestures. Gesture recognition is a classification task, which is a subset of pattern recognition. The fundamental basis of recognition lies in the theory of pattern recognition.This publication focuses on the study of supervised learning methods, deep learning techniques, classification tasks, and gesture recognition trained on proprietary datasets (images obtained and divided into frames from video sequences captured by a webcam or mobile device). These methods significantly expand the range of tasks that can be effectively solved in real-time within the field of gesture recognition.The study presents results using metrics for testing deep learning models. The recognition accuracy of these models has been demonstrated based on precision, recall, F1-score, and support for each class, as well as overall accuracy and average scores. A hybrid architectural neural network model utilizing recurrent and convolutional neural network layers has been tested. Software has been developed to recognize gestures in real-time.

Real-time unsupervised monitoring of earth pressure balance shield-induced sinkholes in mixed-face ground conditions via convolutional variational autoencoders

This study introduces a real-time unsupervised monitoring framework for monitoring sinkhole formation events during earth pressure balance (EPB) shield tunneling operations. A feature extractor (FE) is constructed by coupling variational Autoencoders structure with convolutional neural network layers (VAE-CNN) to manage the complexity of EPB operational data, including non-linearity and temporal dependencies. The monitoring framework consists of two main phases: offline modeling and online monitoring. In the offline modeling phase, an FE model is trained using data-intensive techniques to define a subspace characterizing the behavior of multivariate data without sinkhole formations. The squared prediction error (SPE) statistics and the control limits are computed for detection. During the online monitoring phase, unseen EPB data is propagated to generate SPE values and determine sinkhole events based on whether these values surpass the control limit. Sensor validity index violation counts were used to isolate the most influential variables, while the results demonstrated the superiority of the proposed VAE-CNN method, achieving a 100% detection rate and a 0.9% false alarm rate. The influential variables identified include cutter resolutions per minute, jack speed, screw pressure, torque, and cutter seal components. The monitoring system shows great potential for early warnings during EPB operations to mitigate sinkhole formation risks.

Comparing and Improving Active Learning Uncertainty Measures for Transformer Models by Discarding Outliers

AbstractDespite achieving state-of-the-art results in nearly all Natural Language Processing applications, fine-tuning Transformer-encoder based language models still requires a significant amount of labeled data to achieve satisfying work. A well known technique to reduce the amount of human effort in acquiring a labeled dataset is Active Learning (AL): an iterative process in which only the minimal amount of samples is labeled. AL strategies require access to a quantified confidence measure of the model predictions. A common choice is the softmax activation function for the final Neural Network layer. In this paper, we compare eight alternatives on seven datasets and show that the softmax function provides misleading probabilities. Our finding is that most of the methods primarily identify hard-to-learn-from samples (commonly called outliers), resulting in worse than random performance, instead of samples, which actually reduce the uncertainty of the learned language model. As a solution, this paper proposes Uncertainty-Clipping, a heuristic to systematically exclude samples, which results in improvements for most methods compared to the softmax function.

GeneWorker: An end-to-end robotic reinforcement learning approach with collaborative generator and worker networks

Reinforcement learning aided by the skill conception exhibits potent capabilities in guiding autonomous agents toward acquiring meaningful behaviors. However, in the current landscape of reinforcement learning, a skill is often merely a rudimentary abstraction of a sequence of primitive actions, serving as a component of the input to policy networks with fixed network parameters. This rigid methodology presents obstacles when attempting to integrate with burgeoning techniques such as meta-learning and large language models. To address this issue, we introduce a unique neural skill representation that abstracts the activation of neurons in each neural layer. Based on this, a novel end-to-end robotic reinforcement learning algorithm is proposed, in which two sub-networks, i.e., generator and worker networks, implement collaborative inferences via neural skills. Specifically, the generator produces a series of multi-spatial neural skills, providing efficient guidance for subsequent decision-making; by integrating these skills, the worker can determine its own network weights and biases to cope with various environmental conditions. Therefore, actions can be sampled with flexibly changeable network parameters through the collaboration between generator and worker networks. The experiments demonstrate that GeneWorker can achieve a mean success rate of over 90.67% on continuous robotic tasks and outperforms previous state-of-the-art methods by a minimum of 54% on the pick-and-place task.

Research on deep-sea terrain refinement methods based on satellite remote sensing data

Bathymetric data is crucial for navigation safety and various applications. However, only a small portion of the seafloor has been systematically mapped through direct measurement. The remaining bathymetry is estimated from satellite altimeter data, providing only an approximate representation of the seafloor's shape. To address this issue, the convolutional neural network (CNN) is introduced for the analysis of seabed topography. To the best of our knowledge, this marks the first attempt to incorporate the CNN into the definitive map of the world ocean floor. By employing skip connections based on residual learning, we construct a deeper CNN layer that effectively reduces training complexity. The deeper convolutional neural network layer is used to create a more complex nonlinear mapping model, which is then employed to downscale low-resolution seabed topographic data in order to obtain high-precision seabed topographic information. This process enhances the representation of seabed topographic data to reflect the actual terrain more accurately. The results are compared with those obtained using the bicubic difference method. Conducting tests near the Gulf of Thailand, where Malaysia Airlines Flight MH370 was lost, the results demonstrate that the method proposed in this paper achieves higher accuracy in processing the downscaling of seabed topographic data.

Deep neural network accelerated-group African vulture optimization algorithm for unit commitment considering uncertain wind power

The unit commitment (UC) of power systems serves to plan out the starting and shutdown of units and the power generation of units for a future period of time. However, the UC problem for large-scale systems faces the problems of long solution time and insufficiently accurate solution. Uncertainty in wind power generation poses a great challenge in solving the unit combination problem. This work proposes a deep neural network accelerated-double layer optimization method (DNNA-DOM) to improve the solution efficiency, accuracy, and economy of UC. The outer layer of DNNA-DOM utilizes the proposed deep neural network accelerated-group African vulture optimization algorithm (DNNA-GAVOA) to optimize the on-off condition of conventional coal-fired power units. The inner layer of DNNA-DOM adopts quadratic programming to optimize the solution of the economic dispatch problem of load distribution. The GAVOA optimizes the simulated African vultures in four groups to obtain the optimal solution after diverse exploration, exploitation, and exploitation activities, with low complexity and high accuracy. This work first evaluates the performance of GAVOA by solving seven unimodal functions. Furthermore, the 10-unit simulation by incorporating wind power curves and related constraints is optimized through the DNNA-DOM. The results show that GAVOA outperforms the African vulture optimization algorithm (AVOA) and traditional metaheuristics like the particle swarm algorithm and gray wolf algorithm in terms of lower operating cost and optimization stability. The DNNA-DOM combined with DNNA-GAVOA in the outer layer results in an average daily cost reduction of $36.30 and a 45.1997 % speed increase compared to AVOA in the outer layer.

Design of English Lexical Analysis System for Machine Translation Based on Multimodal Neural Network

Information resources have become a very important source of wealth, and due to differences in language and writing, technological information exchange between countries has become difficult. In this article, we propose a data augmentation method based on statistical models. A framework for a multimodal output model is proposed, taking into account the semantic relevance and importance between different languages. This framework is based on text sequence to sequence framework, decoupled, and a network architecture based on dual stream attention mechanism is designed. A multimodal interactive neural network layer was added between the encoder and decoder, achieving the fusion of multilingual domain information. It compensates for the influence of text on traditional translation generation systems in the process of English Chinese translation, and solves the problems of inaccurate translation results and low similarity with the original text in English Chinese machine translation. The system proposed in this article mainly consists of preprocessing module, lexical analysis and segmentation module, part of speech tagging and phrase analysis module, translation rule construction module, decoding module, translation generation module, etc. The experimental results show that using an improved generation system, compared with traditional generation systems, improves the accuracy of translation generation, has certain advantages, and is more practical.

An analysis of urban sprawl growth and prediction using remote sensing and machine learning techniques

This study examines urban growth to support sustainability by investigating urban sprawl in the Vila Velha Urban Agglomeration (UA). If the spatial area of Vila Velha UA increases by 24.36%, from 22.98 km2 in 1994 to 51.64 km2 in 2024, urban sprawl may affect surrounding areas. Landsat satellite images from 1994, 2004, 2014, and 2024 were analyzed to study the spatiotemporal pattern of urban growth. Shannon's entropy index identified urban sprawl in Vila Velha UA. The QGIS software's MOLUSCE (Modules for Land Use Change Simulations) plug-in, utilizing Cellular Automata (CA) and Artificial Neural Network-Multi Layer Perceptron (ANN-MLP) models, forecasted urban expansion for 2034. The built-up area is projected to increase from 51.64 km2 in 2024 to 62.31 km2 in 2034. Shannon's entropy index indicated a high rate of urban sprawl. Remote sensing and machine learning techniques are crucial for understanding spatial trends, forecasting future expansion, and informing sustainable urban planning. These methods facilitate an accurate assessment of urbanization dynamics, aiding in developing strategies to manage and mitigate sprawl's adverse impacts. This study offers a unique approach to defining urban growth, emphasizing the need for zoning regulations, green space conservation, and infrastructure development to curb urban sprawl and promote sustainable growth in Vila Velha.

A parallel hybrid neural networks model for forecasting returns with candlestick technical trading strategy

Few literature comprehensively addresses the process of input engineering for candlestick price chart and the optimization of model structures as an entire process. Inspired by investors’ decision-making process on the reversal candlestick patterns, where trend classification is separated from candlestick pattern recognition, we propose a novel Parallel Hybrid Neural Networks model (PHNN) designed to enhance return prediction with a candlestick technical trading strategy. This model comprises two distinct sub-networks capable of accommodating various configurations like CNN or LSTM. The final prediction is generated through a fully connected neural layer that integrates the outputs from these sub-networks. To improve the accuracy of the sub-networks, we train the PHNN on prices’ components decomposed through Empirical Mode Decomposition to obtain more precise sub-networks that each one excels in recognition of the trend and the candlestick patterns respectively. Our empirical study provides substantial support for the effectiveness of the PHNN, demonstrating its superiority over benchmark models. Moreover, our results strongly suggest that a double LSTM configuration is the optimal architectural choice for the PHNN.

MambaNet: A Hybrid Neural Network for Predicting the NBA Playoffs

In this paper, we present MambaNet: a hybrid neural network for predicting the outcomes of Basketball games. Contrary to other studies, which focus primarily on season games, this study investigates playoff games. MambaNet is a hybrid neural network architecture that processes a time series of teams’ and players’ game statistics and generates the probability of a team winning or losing an NBA playoff match. In our approach, we utilize Feature Imitating Networks to provide latent signal-processing feature representations of game statistics to further process with convolutional, recurrent, and dense neural layers. Three experiments using five different datasets are conducted to evaluate the performance and generalizability of our architecture against a wide range of previous studies. Our final method successfully predicted the AUC from 0.72 to 0.82, beating the best-performing baseline models by a considerable margin.

An end-to-end multi-task motor imagery EEG classification neural network based on dynamic fusion of spectral-temporal features

Electroencephalograph (EEG) brain-computer interfaces (BCI) have potential to provide new paradigms for controlling computers and devices. The accuracy of brain pattern classification in EEG BCI is directly affected by the quality of features extracted from EEG signals. Currently, feature extraction heavily relies on prior knowledge to engineer features (for example from specific frequency bands); therefore, better extraction of EEG features is an important research direction. In this work, we propose an end-to-end deep neural network that automatically finds and combines features for motor imagery (MI) based EEG BCI with 4 or more imagery classes (multi-task). First, spectral domain features of EEG signals are learned by compact convolutional neural network (CCNN) layers. Then, gated recurrent unit (GRU) neural network layers automatically learn temporal patterns. Lastly, an attention mechanism dynamically combines (across EEG channels) the extracted spectral-temporal features, reducing redundancy. We test our method using BCI Competition IV-2a and a data set we collected. The average classification accuracy on 4-class BCI Competition IV-2a was 85.1 % ± 6.19 %, comparable to recent work in the field and showing low variability among participants; average classification accuracy on our 6-class data was 64.4 % ± 8.35 %. Our dynamic fusion of spectral-temporal features is end-to-end and has relatively few network parameters, and the experimental results show its effectiveness and potential.

Dual Stream Long Short-Term Memory Feature Fusion Classifier for Surface Electromyography Gesture Recognition.

Gesture recognition using electromyography (EMG) signals has prevailed recently in the field of human-computer interactions for controlling intelligent prosthetics. Currently, machine learning and deep learning are the two most commonly employed methods for classifying hand gestures. Despite traditional machine learning methods already achieving impressive performance, it is still a huge amount of work to carry out feature extraction manually. The existing deep learning methods utilize complex neural network architectures to achieve higher accuracy, which will suffer from overfitting, insufficient adaptability, and low recognition accuracy. To improve the existing phenomenon, a novel lightweight model named dual stream LSTM feature fusion classifier is proposed based on the concatenation of five time-domain features of EMG signals and raw data, which are both processed with one-dimensional convolutional neural networks and LSTM layers to carry out the classification. The proposed method can effectively capture global features of EMG signals using a simple architecture, which means less computational cost. An experiment is conducted on a public DB1 dataset with 52 gestures, and each of the 27 subjects repeats every gesture 10 times. The accuracy rate achieved by the model is 89.66%, which is comparable to that achieved by more complex deep learning neural networks, and the inference time for each gesture is 87.6 ms, which can also be implied in a real-time control system. The proposed model is validated using a subject-wise experiment on 10 out of the 40 subjects in the DB2 dataset, achieving a mean accuracy of 91.74%. This is illustrated by its ability to fuse time-domain features and raw data to extract more effective information from the sEMG signal and select an appropriate, efficient, lightweight network to enhance the recognition results.

Intelligent Dust Monitoring System Based on IoT

Dust is a frequent contributor to health risks and changes in the climate, one of the most dangerous issues facing people today. Desertification, drought, agricultural practices, and sand and dust storms from neighboring regions bring on this issue. Deep learning (DL) long short-term memory (LSTM) based regression was a proposed solution to increase the forecasting accuracy of dust and monitoring. The proposed system has two parts to detect and monitor the dust; at the first step, the LSTM and dense layers are used to build a system using to detect the dust, while at the second step, the proposed Wireless Sensor Networks (WSN) and Internet of Things (IoT) model is used as a forecasting and monitoring model. The experiment DL system train and test part was applied to dust phenomena historical data. Its data has been collected through the Iraqi Meteorological Organization and Seismology (IMOS) raw dataset with 170237 of 17023 rows and 10 columns. The LSTM model achieved small time, computationally complexity of, and layers number while being effective and accurate for dust prediction. The simulation results reveal that the model's mean square error test reaches 0.12877 and Mean Absolute Error (MAE) test is 0.07411 at the same rates of learning and exact features values of vector in the dense layer, representing a neural network layer deeply is connected to the LSTM training proposed model. Finally, the model suggested enhances monitoring performance.

The Adjustment Analysis Method of the Active Surface Antenna Based on Convolutional Neural Network

Active surface technique is one of the key technologies to ensure the reflector accuracy of the millimeter/submillimeter wave large reflector antenna. The antenna is complex, large-scale, and high-precision equipment, and its active surfaces are affected by various factors that are difficult to comprehensively deal with. In this paper, based on the advantage of the deep learning method that can be improved through data learning, we propose the active adjustment value analysis method of large reflector antenna based on deep learning. This method constructs a neural network model for antenna active adjustment analysis in view of the fact that a large reflector antenna consists of multiple panels spliced together. Based on the constraint that a single actuator has to support multiple panels (usually 4), an autonomously learned neural network emphasis layer module is designed to enhance the adaptability of the active adjustment neural network model. The classical 8-meter antenna is used as a case study, the actuators have a mean adjustment error of 0.00252 mm, and the corresponding antenna surface error is 0.00523 mm. This active adjustment result shows the effectiveness of the method in this paper.

Unsteady flow-field forecasting leveraging a hybrid deep-learning architecture

With the rapid development of data-driven technology, deep learning has been widely used to study unsteady flow phenomena, owing to its high-dimensional, nonlinear, and excellent big-data processing capabilities. Rapidly and accurately predicting unsteady flow fields has been a major challenge in fluid mechanics. Therefore, we designed a new U-shaped hybrid deep neural network (UDNN) framework using a multilayer convolution. Through the multilayer learning interaction of high-dimensional flow-field data, the temporal and spatial characteristics of the flow-field flow are captured, and the flow-field characteristics are predicted in an end-to-end form. The UDNN comprises a convolutional neural layer, deconvolutional layer, convolutional long-term and short-term layers, and attention-mechanism layer. First, based on computational fluid dynamics, we generated unsteady flow-field datasets of the flow around fixed and rotating cylinders at different Reynolds numbers, which were used as training samples for the network framework. Second, we designed a U-shaped convolutional layer, added horizontal time-series feature processing and attention-mechanism units, and fused the deep feature information predicted by the model with shallow semantic information to predict the flow-field features. In addition, we compared the UDNN, proper orthogonal decomposition – long short-term memory, and traditional convolutional autoencoder – long short-term memory models in terms of the flow-field prediction error, model training time, and inference speed. The final results showed that the proposed UDNN framework achieved high accuracy and strong robustness in predicting unsteady flow fields.

Stress-Level Detection Using RepVGG Neural Network

Stress used to impact our mental and physical well-being,productivity,and overall quality of life. Detecting stress accurately is vital for timely intervention and effective management. In this study, we introduce a new method for detecting stress levels using the RepVGG deep learning architecture. RepVGG stands out for its efficient performance and straightforward structure, making it ideal for analyzing physiological signals and other stress indicators. We using standard metrics to calculate the things like accuracy, precision, recall and etc. Ourfindings reveal that the RepVGG-based method excels in detecting stress levels, surpassing many traditional methods and other deep learning models. Moreover, the model shows strong generalization capabilities across various datasets and conditions. This research underscores the potential of advanced deep learning models like RepVGG in stress detection, opening doors for real-time, scalable, and precise stress monitoring systems. Looking ahead, weaim to integrate this model into wearable devices andmobile apps, enabling continuous stress monitoring and offering personalized stress management advice. Some approach utilizes a rich dataset consists of signals such (HRV) and (GSR), along with other relevant biomarkers. To ensure the model's robustness, we preprocess and augment this data. We then train the RepVGG architecture on this dataset, harnessing its Neural layers for feature extraction and its unique re-parameterizable design for efficient use.

CRPIM: An efficient compute-reuse scheme for ReRAM-based Processing-in-Memory DNN accelerators

Resistive random access memory (ReRAM) is a promising technology for AI Processing-in-Memory (PIM) hardware because of its compatibility with CMOS, small footprint, and ability to complete matrix–vector multiplication workloads inside the memory device itself. However, redundant computations are brought on by duplicate weights and inputs when an MVM has to be split into smaller-granularity sequential sub-works in the real world. Recent studies have proposed repetition-pruning to address this issue, but the buffer allocation strategy for enhancing buffer device utilization remains understudied. In preliminary experiments observing input patterns of neural layers with different datasets, the similarity of repetition allows us to transfer the buffer allocation strategy obtained from a small dataset to the computation with a large dataset. Hence, this paper proposes a practical compute-reuse mechanism for ReRAM-based PIM, called CRPIM, which replaces repetitive computations with buffering and reading. Moreover, the subsequent buffer allocation problem is resolved at both inter-layer and intra-layer levels. Our experimental results demonstrate that CRPIM significantly reduces ReRAM cells and execution time while maintaining adequate buffer and energy overhead.

Adaptive physics-encoded graph neural network for health stage assessment of liquid-propellant rocket engines

The improvement of reliable health monitoring system for liquid-propellant rocket engines (LREs) is a crucial part for reusable launch vehicle, which contributes to providing competitive and cost-effective propulsion systems. Thus, it accentuates the need for reliable and quick health stage assessment of system and follow-up damage-mitigating control. In this paper, we propose a novel adaptive physics-encoded graph neural network for health stage assessment of LREs. Our approach embeds the relations of different sensors obtained through expert experience, which contributes to constructing a physical graph layer. To better capture the information contained in all the sensor data, a novel convolutional layer of adaptive auto-regressive moving average filters is designed, which considers the personalized information propagation needs of each neural network layer. The performance of the proposed method is quantified with data obtained from physics simulations and real-world engineering systems. The results show that our model has potential applicability for the health stage assessment of LREs with high accuracy.

Mechanism-based organization of neural networks to emulate systems biology and pharmacology models.

Deep learning neural networks are often described as black boxes, as it is difficult to trace model outputs back to model inputs due to a lack of clarity over the internal mechanisms. This is even true for those neural networks designed to emulate mechanistic models, which simply learn a mapping between the inputs and outputs of mechanistic models, ignoring the underlying processes. Using a mechanistic model studying the pharmacological interaction between opioids and naloxone as a proof-of-concept example, we demonstrated that by reorganizing the neural networks' layers to mimic the structure of the mechanistic model, it is possible to achieve better training rates and prediction accuracy relative to the previously proposed black-box neural networks, while maintaining the interpretability of the mechanistic simulations. Our framework can be used to emulate mechanistic models in a large parameter space and offers an example on the utility of increasing the interpretability of deep learning networks.

Blind source separation algorithm for noisy hydroacoustic signals based on decoupled convolutional neural networks

Wireless communication technology has been widely used in marine engineering, marine ranching and Marine environmental monitoring. However, structural redundancy and functional confusion exist in applying neural networks in signal separation technology in underwater communication environments, which can result in a slower rate of signal separation and lead to confusion of parameters during transfer learning. Based on this, an end-to-end, internal functionally structured decoupled neural network (D-CNN) blind source separation (BSS) model is proposed in this paper, which can realize a neural network BSS algorithm with a well-defined structure and function. The one-dimensional convolutional neural network layer is used in algorithm to automatically extract observed signal's features, based on the features, and there are two generation modules of separation matrix and scaling coefficients. Then the two modules can be used to separate the observed signal and adjust the signal coefficients to obtain the separated signal. Finally the transfer learning technique is used to generalize the model, which reduces the transfer cost of the model in different application scenarios. Experimental results show that when the communication distance is set to 0.02 km–2 km, the MSE of independent signal and related signal can be reduced by 14.24% and 14.95% respectively compared with the nearest Neural FCA algorithm. The results prove that the proposed algorithm can accurately estimate the source signal and improve the signal reception quality.