Deep Feature Extraction Research Articles

MoRF_ESM: Prediction of MoRFs in disordered proteins based on a deep transformer protein language model.

Molecular recognition features (MoRFs) are particular functional segments of disordered proteins, which play crucial roles in regulating the phase transition of membrane-less organelles and frequently serve as central sites in cellular interaction networks. As the association between disordered proteins and severe diseases continues to be discovered, identifying MoRFs has gained growing significance. Due to the limited number of experimentally validated MoRFs, the performance of existing MoRF's prediction algorithms is not good enough and still needs to be improved. In this research, we present a model named MoRF_ESM, which utilizes deep-learning protein representations to predict MoRFs in disordered proteins. This approach employs a pretrained ESM-2 protein language model to generate embedding representations of residues in the form of attention map matrices. These representations are combined with a self-learned TextCNN model for feature extraction and prediction. In addition, an averaging step was incorporated at the end of the MoRF_ESM model to refine the output and generate final prediction results. In comparison to other impressive methods on benchmark datasets, the MoRF_ESM approach demonstrates state-of-the-art performance, achieving [Formula: see text] higher AUC than other methods when tested on TEST1 and achieving [Formula: see text] higher AUC than other methods when tested on TEST2. These results imply that the combination of ESM-2 and TextCNN can effectively extract deep evolutionary features related to protein structure and function, along with capturing shallow pattern features located in protein sequences, and is well qualified for the prediction task of MoRFs. Given that ESM-2 is a highly versatile protein language model, the methodology proposed in this study can be readily applied to other tasks involving the classification of protein sequences.

SERT-StructNet: Protein secondary structure prediction method based on multi-factor hybrid deep model

Protein secondary structure prediction (PSSP) is a pivotal research endeavour that plays a crucial role in the comprehensive elucidation of protein functions and properties. Current prediction methodologies are focused on deep-learning techniques, particularly focusing on multi-factor features. Diverging from existing approaches, in this study, we placed special emphasis on the effects of amino acid properties and protein secondary structure propensity scores (SSPs) on secondary structure during the meticulous selection of multi-factor features. This differential feature-selection strategy results in a distinctive and effective amalgamation of the sequence and property features. To harness these multi-factor features optimally, we introduced a hybrid deep feature extraction model. The model initially employs mechanisms such as dilated convolution (D-Conv) and a channel attention network (SENet) for local feature extraction and targeted channel enhancement. Subsequently, a combination of recurrent neural network variants (BiGRU and BiLSTM), along with a transformer module, was employed to achieve global bidirectional information consideration and feature enhancement. This approach to multi-factor feature input and multi-level feature processing enabled a comprehensive exploration of intricate associations among amino acid residues in protein sequences, yielding a Q3 accuracy of 84.9% and an Sov score of 85.1%. The overall performance surpasses that of the comparable methods. This study introduces a novel and efficient method for determining the PSSP domain, which is poised to deepen our understanding of the practical applications of protein molecular structures.

DRFNet: a deep radiomic fusion network for nAMD/PCV differentiation in OCT images.

Objective.Neovascular age-related macular degeneration (nAMD) and polypoidal choroidal vasculopathy (PCV) present many similar clinical features. However, there are significant differences in the progression of nAMD and PCV. and it is crucial to make accurate diagnosis for treatment. In this paper, we propose a structure-radiomic fusion network (DRFNet) to differentiate PCV and nAMD in optical coherence tomography (OCT) images.Approach.The subnetwork (RIMNet) is designed to automatically segment the lesion of nAMD and PCV. Another subnetwork (StrEncoder) is designed to extract deep structural features of the segmented lesion. The subnetwork (RadEncoder) is designed to extract radiomic features from the segmented lesions based on radiomics. 305 eyes (155 with nAMD and 150 with PCV) are included and manually annotated CNV region in this study. The proposed method was trained and evaluated by 4-fold cross validation using the collected data and was compared with the advanced differentiation methods.Main results.The proposed method achieved high classification performace of nAMD/PCV differentiation in OCT images, which was an improvement of 4.68 compared with other best method.Significance. The presented structure-radiomic fusion network (DRFNet) has great performance of diagnosing nAMD and PCV and high clinical value by using OCT instead of indocyanine green angiography.

DDFC: deep learning approach for deep feature extraction and classification of brain tumors using magnetic resonance imaging in E-healthcare system

This research explores the use of gated recurrent units (GRUs) for automated brain tumor detection using MRI data. The GRU model captures sequential patterns and considers spatial information within individual MRI images and the temporal evolution of lesion characteristics. The proposed approach improves the accuracy of tumor detection using MRI images. The model’s performance is benchmarked against conventional CNNs and other recurrent architectures. The research addresses interpretability concerns by employing attention mechanisms that highlight salient features contributing to the model’s decisions. The proposed model attention-gated recurrent units (A-GRU) results show promising results, indicating that the proposed model surpasses the state-of-the-art models in terms of accuracy and obtained 99.32% accuracy. Due to the high predictive capability of the proposed model, we recommend it for the effective diagnosis of Brain tumors in the E-healthcare system.

Multichannel consecutive data cross-extraction with 1DCNN-attention for diagnosis of power transformer

Power transformer plays a critical role in grid infrastructure, and its diagnosis is paramount for maintaining stable operation. However, the current methods for transformer diagnosis focus on discrete dissolved gas analysis, neglecting deep feature extraction of multichannel consecutive data. The unutilized sequential data contains the significant temporal information reflecting the transformer condition. In light of this, the structure of multichannel consecutive data cross-extraction (MCDC) is proposed in this article in order to comprehensively exploit the intrinsic characteristic and evaluate the states of transformer. Moreover, for the better accommodation in scenario of transformer diagnosis, one dimensional convolution neural network attention (1DCNN-attention) mechanism is introduced and offers a more efficient solution given the simplified spatial complexity. Finally, the effectiveness of MCDC and the superior generalization ability, compared with other algorithms, are validated in experiments conducted on a dataset collected from real operation cases of power transformer. Additionally, the better stability of 1DCNN-attention has also been certified.

Deep Reinforcement Learning Model for Stock Portfolio Management Based on Data Fusion

Deep reinforcement learning (DRL) can be used to extract deep features that can be incorporated into reinforcement learning systems to enable improved decision-making; DRL can therefore also be used for managing stock portfolios. Traditional methods cannot fully exploit the advantages of DRL because they are generally based on real-time stock quotes, which do not have sufficient features for making comprehensive decisions. In this study, in addition to stock quotes, we introduced stock financial indices as additional stock features. Moreover, we used Markowitz mean-variance theory for determining stock correlation. A three-agent deep reinforcement learning model called Collaborative Multi-agent reinforcement learning-based stock Portfolio management System (CMPS) was designed and trained based on fused data. In CMPS, each agent was implemented with a deep Q-network to obtain the features of time-series stock data, and a self-attention network was used to combine the output of each agent. We added a risk-free asset strategy to CMPS to prevent risks and referred to this model as CMPS-Risk Free (CMPS-RF). We conducted experiments under different market conditions using the stock data of China Shanghai Stock Exchange 50 and compared our model with the state-of-the-art models. The results showed that CMPS could obtain better profits than the compared benchmark models, and CMPS-RF was able to accurately recognize the market risk and achieved the best Sharpe and Calmar ratios. The study findings are expected to aid in the development of an efficient investment-trading strategy.

Novel cloud-AIoT fault diagnosis for industrial diesel generators based hybrid deep learning CNN-BGRU algorithm

The industrial diesel generator is essential to provide electric power during emergencies and requires accurate failure diagnosis in preventative maintenance services. This research presents a novel AIoT system-based hybrid deep learning algorithm for online monitoring of the anomaly conditions of 125 kW/250 kW industrial generators. Firstly, the IoT modules with different precise sensors are developed to be installed on 125 kW and 250 kW industrial generators to simulate other faults conditions in the laboratory. Then, the supervised learning operations are deployed to train deep learning algorithms with collected labeled data. The proposed algorithm, a hybrid convolution neural network bidirectional gate recurrent unit (CNN-BGRU), is deployed to extract deep features from long sequential historical signals and classify the anomaly conditions. The experiments in different 10 s, 20 s, and 30 s sampling frequencies are performed to evaluate and analyze with various benchmarks, which demonstrate efficiencies of the proposed CNN-BGRU method with other traditional deep learning algorithms, including RNN, CNN, GRU, LSTM, BRNN, and BGRU methods. The proposed hybrid method accomplishes the supreme augmentation of 29.43 % loss, 36.55 % validating loss, 25.86 % MAE, 27.00 % validating MAE, 16.78 % MSE, 20.64 % validating MSE, 6.36 % Pre, 25.34 % validating Pre, 17.30 % Rec, and 26.91 % validating Rec compared with state-of-the-art approaches. The AIoT system-based hybrid CNN-BGRU algorithm provides better improvement and higher accuracy in fault diagnosis of firefighting pumps for maintenance service in Industry 4.0.

Cuffless blood pressure estimation from photoplethysmography using deep convolutional neural network and transfer learning

Blood pressure (BP) monitoring is an essential indicator for diseases of the cardiovascular system. Early detection of abnormalities in BP helps to significantly reduce the risk of diseases such as chronic heart failure, stroke and hypertension. In this study, we propose a new framework for systolic, diastolic and mean arterial blood pressure (SBP, DBP and MBP) estimation using PPG signals and its derivatives. The proposed framework includes the extraction of deep features by pre-trained CNN models based on transfer learning using images of signal waveforms. The most distinctive features are obtained by using the RFE feature selection algorithm on the extracted deep features. The selected deep features are exposed to a range of well-known machine and deep learning regression methods. The proposed BP estimation models have been evaluated with statistical metrics, visual analytical tools and international gold standards such as AAMI and BHS. Experimental results show that the first derivative of PPG, the VPG input image, the deep features obtained with DenseNet121 and the bi-directional gated recurrent unit (Bi-GRU) algorithm provide the best BP estimation performance. The results reveal that the SBP, DBP and MBP estimation models are grade-A according to the BHS protocol and meet the AAMI standard. The presented framework has been compared with well-established studies using the MIMIC II dataset, and the comparative analysis confirms that the proposed method outperforms existing techniques. The deep learning model for BP estimation offers a highly accurate, fast and practical system, especially in the monitoring of patients at risk of hypertension.

Identification of multiclass tympanic membranes by using deep feature transfer learning and hyperparameter optimization

Middle ear health is a process that generally depends on eardrum health. Middle ear disorders are more common during childhood. Permanent damage may occur in bacterial or viral infections in this region if an early diagnosis is not made. Infectious ear disease, especially known as Otitis Media, is one of the diseases. In this study, morphological features of images are obtained by using various feature extraction methods. Deep feature-based transfer learning and hyperparameter optimization methods were used to detect the presence and type of otitis media. While the EfficientNet convolutional neural network (CNN) model was used to extract deep features, KNN, SVM, and Ensemble classifiers were used as classifiers. Bayesian, Grid Search, and Random Search were used for hyperparameter optimization. As a result of the experiments carried out, it was observed that the classification performance was 99.1%.

Hyperspectral Image Classification with the Orthogonal Self-Attention ResNet and Two-Step Support Vector Machine

Hyperspectral image classification plays a crucial role in remote sensing image analysis by classifying pixels. However, the existing methods require more spatial–global information interaction and feature extraction capabilities. To overcome these challenges, this paper proposes a novel model for hyperspectral image classification using an orthogonal self-attention ResNet and a two-step support vector machine (OSANet-TSSVM). The OSANet-TSSVM model comprises two essential components: a deep feature extraction network and an improved support vector machine (SVM) classification module. The deep feature extraction network incorporates an orthogonal self-attention module (OSM) and a channel attention module (CAM) to enhance the spatial–spectral feature extraction. The OSM focuses on computing 2D self-attention weights for the orthogonal dimensions of an image, resulting in a reduced number of parameters while capturing comprehensive global contextual information. In contrast, the CAM independently learns attention weights along the channel dimension. The CAM autonomously learns attention weights along the channel dimension, enabling the deep network to emphasise crucial channel information and enhance the spectral feature extraction capability. In addition to the feature extraction network, the OSANet-TSSVM model leverages an improved SVM classification module known as the two-step support vector machine (TSSVM) model. This module preserves the discriminative outcomes of the first-level SVM subclassifier and remaps them as new features for the TSSVM training. By integrating the results of the two classifiers, the deficiencies of the individual classifiers were effectively compensated, resulting in significantly enhanced classification accuracy. The performance of the proposed OSANet-TSSVM model was thoroughly evaluated using public datasets. The experimental results demonstrated that the model performed well in both subjective and objective evaluation metrics. The superiority of this model highlights its potential for advancing hyperspectral image classification in remote sensing applications.

Determination and investigation of shear wave velocity based on one deep/machine learning technique

Rock physics plays an important role in the oil, gas, and water industries by providing essential data for reservoir management. This study focuses on predicting Shear Wave Velocity (Vs), a key geophysical parameter, using the Deep Belief Network (DBN) and Random Forest (RF) algorithms. The dataset is divided for training, validation, and testing, revealing that the DBN model surpasses the RF and empirical models in accuracy. With a low RMSE value of 0.0398 km/s in testing and an R2 of 0.9775, the DBN model demonstrates precision in Vs prediction. The analysis identifies specific parameters—Measure Depth (MD), Caliper (CALL), Gamma Ray (GR), Shallow Electrical Resistivity (RES-S), Medium Electrical Resistivity (RES-M), Deep Electrical Resistivity (RES-D), Compressional Wave Velocity (DCT), Neutron Porosity (NPHI), and Photoelectric Coefficient Index (PEF)—as significantly influencing Vs, while CALI and GR have a relatively lesser effect. The results of the Vs prediction indicated that the DBN algorithm excels in deep feature extraction, unsupervised pretraining, and efficient handling of complex data. Ultimately, the DBN algorithm proves to be a superior and time-saving alternative for Vs prediction and analysis in subsurface reservoirs.

Identification of leaf diseases in field crops based on improved ShuffleNetV2.

Rapid and accurate identification and timely protection of crop disease is of great importance for ensuring crop yields. Aiming at the problems of large model parameters of existing crop disease recognition methods and low recognition accuracy in the complex background of the field, we propose a lightweight crop leaf disease recognition model based on improved ShuffleNetV2. First, the repetition number and the number of output channels of the basic module of the ShuffleNetV2 model are redesigned to reduce the model parameters to make the model more lightweight while ensuring the accuracy of the model. Second, the residual structure is introduced in the basic feature extraction module to solve the gradient vanishing problem and enable the model to learn more complex feature representations. Then, parallel paths were added to the mechanism of the efficient channel attention (ECA) module, and the weights of different paths were adaptively updated by learnable parameters, and then the efficient dual channel attention (EDCA) module was proposed, which was embedded into the ShuffleNetV2 to improve the cross-channel interaction capability of the model. Finally, a multi-scale shallow feature extraction module and a multi-scale deep feature extraction module were introduced to improve the model's ability to extract lesions at different scales. Based on the above improvements, a lightweight crop leaf disease recognition model REM-ShuffleNetV2 was proposed. Experiments results show that the accuracy and F1 score of the REM-ShuffleNetV2 model on the self-constructed field crop leaf disease dataset are 96.72% and 96.62%, which are 3.88% and 4.37% higher than that of the ShuffleNetV2 model; and the number of model parameters is 4.40M, which is 9.65% less than that of the original model. Compared with classic networks such as DenseNet121, EfficientNet, and MobileNetV3, the REM-ShuffleNetV2 model not only has higher recognition accuracy but also has fewer model parameters. The REM-ShuffleNetV2 model proposed in this study can achieve accurate identification of crop leaf disease in complex field backgrounds, and the model is small, which is convenient to deploy to the mobile end, and provides a reference for intelligent diagnosis of crop leaf disease.

Enhanced process monitoring for industrial coking furnace using a dual‐channel pooling and homologous bilinear model‐based convolutional neural network

AbstractThis paper introduces a process monitoring method using a convolutional neural network (CNN) with dual‐channel pooling and homologous bilinear models (DCP‐HBM). This method aims to enhance the stability of industrial coking furnaces and reduce maintenance costs. It is significantly relevant for addressing challenges such as complexity, high dimensionality, and strong coupling, which are difficult to manage with traditional fault diagnosis methods. In this study, a deep feature extraction module is employed to gather feature information. To effectively reduce the impact of noise and strengthen the correlation between features, DCP is incorporated after deep feature extraction. Concurrently, a HBM is introduced for feature fusion, further refining the characteristics of each state. Faults are then accurately classified through a classification module. The method utilizes DCP to filter and focus input data, thus making the model more attentive to task‐relevant information. This enhances the model's understanding and representation of the input data. The introduction of the HBM further refines the features extracted by the network and increases the precision of feature extraction. This leads to improved recognition of faults with high similarity, thereby enhancing the accuracy of the overall process monitoring. Experimental results demonstrate that this method exhibits strong performance in the monitoring of industrial coking furnaces, indicating its wide‐ranging application prospects.

A Degradation Prediction Method for PEM Fuel Cell Based on Deep Temporal Feature Extraction and Transfer Learning

The increasing environmental issues such as air pollution and energy scarcity of fossil fuels require the acceleration of the electrification in various fields, especially for the transportation. According to this development trend, fuel cell systems gradually become a potential alternative to traditional power systems for high efficiency, high energy density and zero pollution, while the relatively short life time has tremendously limited its large-scale commercial application. To extend the fuel cell lifespan, degradation predictive methods are proved to have outstanding effect. In this article, a hybrid prediction model based on deep temporal feature extraction and transfer learning is proposed. Firstly, the health index is constructed by extracting the fuel cell degradation features through a temporal convolutional network; on this basis, transfer learning is performed according to the feature extraction, and finally the extracted features are input into the long short-term memory model to complete the fuel cell degradation prediction. The effectiveness of the algorithm is verified with experimentally extracting degradation data under different operating conditions, and the results show that the proposed method can effectively improve the prediction accuracy and decrease the computation by extracting key parameters of fuel cell degradation and is independent of operating conditions.

Deep Feature Extraction and Attention Fusion for Multimodal Emotion Recognition

Deep Feature Extraction and Attention Fusion for Multimodal Emotion Recognition

Face-Inception-Net for Recognition

Face recognition in general scenarios has been saturated in recent years, but there is still room to enhance model performance in extreme scenarios and fairness situations. Inspired by the successful application of Transformer and ConvNet in computer vision, we propose a FIN-Block, which gives a more flexible composition paradigm for building a novel pure convolution model and provides a foundation for constructing a new framework for general face recognition in both extreme scenarios and fairness situations. FIN-Block-A uses a combination of stacked large-size convolution kernels and parallel convolution branches to ensure a large spatial receptive field while improving the module’s deep feature embedding and extraction capabilities. FIN-Block-B takes advantage of stacked orthogonal convolution kernels and parallel branches to balance model size and performance. By applying FIN-Block with an adapted convolution kernel size in different stages, we built a reasonable and novel framework Face-Inception-Net, and the performance of the model is highly competitive with ConvNeXt and InceptionNeXt. The models were trained on CASIA-WebFace and MS-wo-RFW databases and evaluated on 14 mainstream benchmarks, including LFW, extreme scene, and fairness test sets. The proposed Face-Inception-Net achieved the highest average TAR@FAR0.001 of 95.9% in all used benchmarks, fully demonstrating effectiveness and generality in various scenarios.

Transfer learning model for cash-instrument prediction adopting a Transformer derivative

Investors aiming for high market returns must accurately predict the prices of various cash instruments. However, making accurate predictions is challenging due to the complex cyclic and trending characteristic of markets, characterized by high volatility and unpredictable fluctuations. Furthermore, many studies overlook how interactions between different markets affect price movements. To address these problems, this research introduces a deep transfer-learning approach derived from the Transformer model, named the rotary-positional encoding autocorrelation Transformer (RAT). Unlike traditional methods, the RAT employs autocorrelation instead of self-attention to more effectively capture periodic features, while rotary-positional encoding preserves both the absolute and relative positioning within sequences to enhance trend understanding. Through transfer learning, the RAT model extracts deep features from a source domain and applies them to a target domain, demonstrating superior performance over LSTM, CNN-LSTM, gated recurrent units (GRUs), and Transformer models in multi-day predictions across 12 cash-instrument datasets. It achieved a substantial increase in accuracy, with a 35.83% reduction in mean squared error (MSE), a 23.95% reduction in mean absolute error (MAE), and a 32.63% increase in the coefficient of determination (R2). This study validates the RAT model’s effectiveness in predicting financial instrument prices.

Deep Learning of Partially Labeled Data for Quality Prediction Based on Stacked Target-Related Laplacian Autoencoder.

Partially labeled data, which is common in industrial processes due to the low sampling rate of quality variables, remains an important challenge in soft sensor applications. In order to exploit the information from partially labeled data, a target-related Laplacian autoencoder (TLapAE) is proposed in this work. In TLapAE, a novel target-related Laplacian regularizer is developed, which aims to extract structure-preserving and quality-related features by preserving the feature-target mapping according to the local geometrical structure of the data. In addition, stacked TLapAE (STLapAE) is further constructed to extract deep feature representations of the data by hierarchically stacking TLapAE blocks. For model training, backward propagation equations are derived based on matrix calculus techniques to update the model parameters of the proposed TLapAE. The effectiveness of the proposed STLapAE is evaluated using the butane content prediction case in a debutanizer column, the silicon content prediction case in a blast furnace (BF) ironmaking process, and the ethane concentration prediction case in an ethylene fractionator. The results show that the proposed TLapAE model has significantly improved prediction accuracy compared to soft sensors using only labeled data and other partially labeled data modeling methods.

Flexible Job Shop Scheduling via Dual Attention Network-Based Reinforcement Learning.

Flexible manufacturing has given rise to complex scheduling problems such as the flexible job shop scheduling problem (FJSP). In FJSP, operations can be processed on multiple machines, leading to intricate relationships between operations and machines. Recent works have employed deep reinforcement learning (DRL) to learn priority dispatching rules (PDRs) for solving FJSP. However, the quality of solutions still has room for improvement relative to that by the exact methods such as OR-Tools. To address this issue, this article presents a novel end-to-end learning framework that weds the merits of self-attention models for deep feature extraction and DRL for scalable decision-making. The complex relationships between operations and machines are represented precisely and concisely, for which a dual-attention network (DAN) comprising several interconnected operation message attention blocks and machine message attention blocks is proposed. The DAN exploits the complicated relationships to construct production-adaptive operation and machine features to support high-quality decision-making. Experimental results using synthetic data as well as public benchmarks corroborate that the proposed approach outperforms both traditional PDRs and the state-of-the-art DRL method. Moreover, it achieves results comparable to exact methods in certain cases and demonstrates favorable generalization ability to large-scale and real-world unseen FJSP tasks.

Two-Level Spatio-Temporal Feature Fused Two-Stream Network for Micro-Expression Recognition.

Micro-expressions, which are spontaneous and difficult to suppress, reveal a person's true emotions. They are characterized by short duration and low intensity, making the task of micro-expression recognition challenging in the field of emotion computing. In recent years, deep learning-based feature extraction and fusion techniques have been widely used for micro-expression recognition, particularly methods based on Vision Transformer that have gained popularity. However, the Vision Transformer-based architecture used in micro-expression recognition involves a significant amount of invalid computation. Additionally, in the traditional two-stream architecture, although separate streams are combined through late fusion, only the output features from the deepest level of the network are utilized for classification, thus limiting the network's ability to capture subtle details due to the lack of fine-grained information. To address these issues, we propose a new two-level spatio-temporal feature fused with a two-stream architecture. This architecture includes a spatial encoder (modified ResNet) for learning texture features of the face, a temporal encoder (Swin Transformer) for learning facial muscle motor features, a feature fusion algorithm for integrating multi-level spatio-temporal features, a classification head, and a weighted average operator for temporal aggregation. The two-stream architecture has the advantage of extracting richer features compared to the single-stream architecture, leading to improved performance. The shifted window scheme of Swin Transformer restricts self-attention computation to non-overlapping local windows and allows cross-window connections, significantly improving the performance and reducing the computation compared to Vision Transformer. Moreover, the modified ResNet is computationally less intensive. Our proposed feature fusion algorithm leverages the similarity in output feature shapes at each stage of the two streams, enabling the effective fusion of multi-level spatio-temporal features. This algorithm results in an improvement of approximately 4% in both the F1 score and the UAR. Comprehensive evaluations conducted on three widely used spontaneous micro-expression datasets (SMIC-HS, CASME II, and SAMM) consistently demonstrate the superiority of our approach over comparative methods. Notably, our approach achieves a UAR exceeding 0.905 on CASME II, making it one of the few frameworks in the published micro-expression recognition literature to achieve such high performance.