Extract Temporal Features Research Articles

STFE-Net: A Spatial-Temporal Feature Extraction Network for Continuous Sign Language Translation

The main challenge of continuous sign language translation (CSLT) lies in the extraction of both discriminative spatial features and temporal features. In this paper, a spatial-temporal feature extraction network (STFE-Net) is proposed for CSLT, which optimally fuses spatial and temporal features, extracted by the spatial feature extraction network (SFE-Net) and the temporal feature extraction network (TFE-Net), respectively. SFE-Net performs pose estimation for the presenters in sign-language videos. Based on COCO-WholeBody, 133 key points are abbreviated to 53 key points, according to the characteristics of the sign language. High-resolution pose estimation is performed on the hands, along with the whole-body pose estimation, to obtain finer-grained hand features. The spatial features extracted by SFE-Net and the sign language words are then fed to TFE-Net, which is based on Transformer with relative position encoding. In this paper, a dataset for Chinese continuous sign language was created and used for evaluation. STFE-Net achieves Bilingual Evaluation Understudy (BLEU-1, BLEU-2, BLEU-3, BLEU-4) scores of 77.59, 75.62, 74.25, 72.14, respectively. Furthermore, our proposed STFE-Net was also evaluated on two public datasets, RWTH-Phoenix-Weather 2014T and CLS. The BLEU-1, BLEU-2, BLEU-3 and BLEU-4 scores achieved by our method on the former dataset are 48.22, 33.59, 26.41 and 22.45, respectively, and the corresponding scores are 61.54, 58.76, 57.93 and 57.52, respectively, on the latter dataset. Experiment results show that our model can achieve promising performance. If any reader needs the code or dataset, please email lunfee@whut.edu.cn.

System Derived Spatial-Temporal CNN for High-Density fNIRS BCI.

An intuitive and generalisable approach to spatial-temporal feature extraction for high-density (HD) functional Near-Infrared Spectroscopy (fNIRS) brain-computer interface (BCI) is proposed, demonstrated here using Frequency-Domain (FD) fNIRS for motor-task classification. Enabled by the HD probe design, layered topographical maps of Oxy/deOxy Haemoglobin changes are used to train a 3D convolutional neural network (CNN), enabling simultaneous extraction of spatial and temporal features. The proposed spatial-temporal CNN is shown to effectively exploit the spatial relationships in HD fNIRS measurements to improve the classification of the functional haemodynamic response, achieving an average F1 score of 0.69 across seven subjects in a mixed subjects training scheme, and improving subject-independent classification as compared to a standard temporal CNN.

Touchformer: A Transformer-Based Two-Tower Architecture for Tactile Temporal Signal Classification.

Haptic temporal signal recognition plays an important supporting role in robot perception. This paper investigates how to improve classification performance on multiple types of haptic temporal signal datasets using a Transformer model structure. By analyzing the feature representation of haptic temporal signals, a Transformer-based two-tower structural model, called Touchformer, is proposed to extract temporal and spatial features separately and integrate them using a self-attention mechanism for classification. To address the characteristics of small sample datasets, data augmentation is employed to improve the stability of the dataset. Adaptations to the overall architecture of the model and the training and optimization procedures are made to improve the recognition performance and robustness of the model. Experimental comparisons on three publicly available datasets demonstrate that the Touchformer model significantly outperforms the benchmark model, indicating our approach's effectiveness and providing a new solution for robot perception.

Deep Learning Localization Algorithm Integrating Attention Mechanism in Database Information Query

This study aims to solve the problems of traditional indoor car search positioning technology in terms of positioning accuracy and functionality. Based on database technology and deep learning technology, an LSTM model with attention mechanism was established. This model can simultaneously extract temporal and spatial features, and use attention mechanism for feature importance recognition. The entire positioning model has been designed as a triple functional entrance that includes positioning, car storage, and reverse car search, enhancing the user's coherent experience. The data results show that the root mean square error of the LSTM (Attention) model designed in the study is 0.216, and the variance is 0.092. Among similar positioning models, the index value is the smallest, while the CDF line rises the fastest and the maximum value is the highest. The research conclusion indicates that the LSTM (Attention) indoor positioning model designed in this study has better computational performance and can help users achieve more accurate positioning and vehicle navigation.

Heterogeneous Recurrence Analysis of imaged-EEG for Spatio-Temporal Epileptic Seizure Detection.

Epilepsy is known as a heterogeneous neurological disorder affecting 1 to 3 percent of the worldwide population. Epileptic seizures occur when brain cells feature abnormal synchronized recurrent activities. In this study, Heterogeneous Recurrence Analysis (HRA) is utilized to investigate seizure phenomena and to develop a spatio-temporal seizure detection algorithm. Using recurrence characteristics of multichannel scalp electroencephalography (EEG) signals we tried to model seizures by frequent feature values in neighboring channels. For this aim, some elementary features in time, frequency, and statistical domain are extracted from 2-second epochs and used to make imaged-EEGs. Each recording's channel-set provides the ground rule for placing feature values in the specified pixels in the imaged-EEGs. These images are then fed to the HRA algorithm to extract heterogeneous recurrence features in each region of the image. Despite existing methods using each individual channel's characteristics as features for each epoch, this method can provide spatial heterogeneous recurrence information for each region of the image, consequently regions of the brain. Using the HRA method with imaged-EEGs also gives us the ability to extract temporal recurrent features from successive epochs. As a result, groups of channels are considered as one region, and their recurrent behaviors in different traits are quantified as HRA features. Our method was evaluated using two publicly available epileptic EEG datasets recorded from pediatric patients at Boston Children's Hospital (CHB-MIT) and American university of Beirut Medical Center (ABMC). Considering only temporal detection of seizures, the averaged evaluation parameters are 99.6% accuracy, 99.7% sensitivity, 99.4% specificity on 24 patients of CHB-MIT dataset, and 98.5% accuracy, 97.9% sensitivity, 98.5% specificity on 6 patients of ABMC. The results show that the accuracy and specificity of the proposed method are comparable to the best machine learning baseline methods while the sensitivity is better. Besides good classification results, HRA on imaged-EEGs can give us valuable information about the patient's brain lobe/s in which recurrent features are distinctive for seizure detection.

Motor Imagery EEG Decoding Based on Multi-scale Hybrid Networks and Feature Enhancement.

Motor Imagery (MI) based on Electroencephalography (EEG), a typical Brain-Computer Interface (BCI) paradigm, can communicate with external devices according to the brain's intentions. Convolutional Neural Networks (CNN) are gradually used for EEG classification tasks and have achieved satisfactory performance. However, most CNN-based methods employ a single convolution mode and a convolution kernel size, which cannot extract multi-scale advanced temporal and spatial features efficiently. What's more, they hinder the further improvement of the classification accuracy of MI-EEG signals. This paper proposes a novel Multi-Scale Hybrid Convolutional Neural Network (MSHCNN) for MI-EEG signal decoding to improve classification performance. The two-dimensional convolution is used to extract temporal and spatial features of EEG signals and the one-dimensional convolution is used to extract advanced temporal features of EEG signals. In addition, a channel coding method is proposed to improve the expression capacity of the spatiotemporal characteristics of EEG signals. We evaluate the performance of the proposed method on the dataset collected in the laboratory and BCI competition IV 2b, 2a, and the average accuracy is at 96.87%, 85.25%, and 84.86%, respectively. Compared with other advanced methods, our proposed method achieves higher classification accuracy. Then we use the proposed method for an online experiment and design an intelligent artificial limb control system. The proposed method effectively extracts EEG signals' advanced temporal and spatial features. Additionally, we design an online recognition system, which contributes to the further development of the BCI system.

Adaptive Fusion Feature Transfer Learning Method For NILM

Non-invasive load monitoring (NILM) aims to extract the power consumption of individual appliances from a smart meter that measures the total power consumption of all appliances. At present, deep learning methods have achieved leading results. However, the need for a large number of training data and the poor generalization ability of models limit the further development of NILM. To break through these limitations, this paper proposes an adaptive fusion feature transfer learning method. Firstly, to provide rich features for feature transfer, multiple feature extraction branches are used to extract temporal and spatial features from different perspectives. These features are fused for adaptive adjustment. Secondly, the attention mechanism is used to adapt the features extracted from the model of the source task so that they are more conducive to the model of the target task. Finally, abundant simulation experiments are performed for appliance transfer and house transfer, respectively. It is verified that the proposed method can achieve better results than the existing method by using only a small amount of training data and retraining a part of the new model.

Temporal series features extraction based on Bi-ConvLSTM of Alzheimer’s disease pridiction CTISS model

目的 阿尔兹海默症疾病的发展是一个缓慢的过程，患者在出现明显症状之后才开始用药，而这时患者的脑损伤已经过于严重难以恢复，阿尔兹海默症的早期预测能够尽早干预这一过程。目前2D及3D的卷积方法大多基于单次检测的磁共振成像（magnetic resonance imaging，MRI）影像进行特征提取，但是病程的发展预测更应该关注时序的特征。考虑到病例在疾病发展过程中的MRI检查时序能够为诊断提供疾病发展的影像变化的有效信息，本文建立CTISS（ConvLSTM based on temporal images series slice）模型，提取病程发展不同阶段的变化特征，实现阿尔兹海默症病病程的早期预测。方法 与现有算法通过一阶段MRI影像提取特征不同，该模型采用两阶段脑影像建立了一种分层时序卷积的网络结构，采用自适应学习率方法RMSprop（root mean square prop）训练模型。参数优化实验结果显示，2层卷积时序含注意力机制，1层长短时记忆含注意力机制的模型结构性能最好。算法采用时序卷积双向长短时记忆模型（bi-directional long short-term memory，Bi-ConvLSTM）及注意力机制，在大脑影像的分层切面上进行时序特征提取。结果 CTISS模型与现有的5种算法在4个分类任务进行对比。结果显示，CTISS模型在长序列阿尔兹海默症病程预测中取得很好的预测性能。CTISS模型能够捕捉与疾病发展过程相关的长时序影像特征，并获得更好的算法性能，特别在3分类任务中AUC（area under curve）较其他算法提升12%的精度。同时，CTISS模型其他深度模型所提取的全脑萎缩空腔增大、大脑白质区域纤维化的影像变化特征与阿尔兹海默症的病理解剖所获得的结论一致。结论 与其他算法相比，CTISS模型的AUC在3分类中取得较2分类更好的算法性能提升，因此CTISS模型在长期跟踪数据集上具有较其他算法更好的表现。;Objective To realize the Alzheimer’s disease（AD）prediction，most of methods are focused on magnetic resonance imaging（MRI）image baseline of the patient. Current 2D and 3D convolutional neural networks（CNNs）are commonly used for MRI feature extraction. But，the challenge issue is required to learn temperal feature of the image changing although CNN deep models can be used on image feature selection. To extract the changing features and get the early prediction of Alzheimer’s disease effectively，we develop a ConvLSTM（convolution long short term memory）based on temporal images series slice（CTISS）model further. Method The CTISS model consist of 1）ConvLSTM and 2）Attention，which can extract spatio-temporal features from MRI and locate the disease relative spatial position of each. After slicing 3D MRI into 2D slices，the CTISS model can extract features from corresponding channels of the slices，one slice for each channel. Parameter optimization of the CTISS model shows that Attention-incorporated Conv2D layer is more efficient than the attention-intergrated bi-directional convolutional long short-term memory（Bi-ConvLSTM）layer. Sequential MRI images are split into slices-consistent，which are categorized into the corresponding channels. Each channel is fed with 2 serial slices. This model can use two stages of brain image beyond one-stage MRI. A network structure of hierarchical and sequential convolution is established，and root mean square prop（RMSprop）is used as the adaptive learning rate for training. To extract temporal features from brain image slices，our algorithm is constructed in terms of Bi-ConvLSTM model with Attention. Result The experiments are equipped with 4 Titan GPU，12 GB memory per GPU，selecting 691 patients，totally 1 765 samples in AD/MCI/NC classes，which have 751 AD，500 MCI and 514 NC. CTISS model is created by Keras2. 0 python library. First，MRI images are normalized into size 256×256×256. Second，each image is sliced into size 256×256 and grouped into time sequence according to the medical case ID. Finally，all those sequential slices are sent to 256 channels separately. Our CTISS model can capture long-term sequential image features related to the status of the disease，which is 0. 89 in AD vs NC，0. 81 in MCI vs NC，0. 72 in AD vs MCI and 0. 94 in AD vs MCI vs NC. Especially，the accuracy can be improved 12% in the 3-facets classification，which are 1）prediction performance， 2）feature extraction ability，and 3）comparative analysis with existing models. At the same time，the CTISS modelextracted image features like the cavity enlargement of the whole-brain atrophy and the regional fibrosis of the brain are consistent with the pathological anatomy of Alzheimer’s disease. Conclusion The AUC（area under curve）of CTISS model has its better optimization in 3 facets-classified than 2 facets-classified. The potentials of the CTISS model is linked to 2 aspects：1）its performance is stable in different size of classes；2）it achieves better AUC in long term series data for the sequential feature learning.

Integrating Piecewise Linear Representation and Deep Learning for Trading Signals Forecasting

Trading signal forecasting is an interesting but challenging research topic in the field of financial investment, since the financial market is an nonlinearity and high volatility system influenced by too many factors, and a small improvement in forecasting performance can bring profits. To realize trading signals detection, this paper presents a novel method which integrates piecewise linear representation (PLR) with a deep learning learning framework to predict the financial trading points. Firstly, we utilize PLR to generate a number of turning points (valleys or peaks) from trading data and formulate the trading points prediction as a three-class classification problem. Then, the framework combined a convolutional neural network (CNN) for spatial features extraction and a long short-term memory (LSTM) network for temporal domain features extraction (CNN-LSTM) is used to learn the prediction model between the trading points and the financial time series data. Finally, we conduct a series of experiments among PLR-CNN-LSTM, PLR-CNN-TA and PLR-LSTM on companies of US, Turkey and daily Exchange-Traded Fund (ETFs) to test the performance of our established method. The experiment results show that our proposed method has better model performance and profitability with different investment strategies.

Multi-Semantic Discriminative Feature Learning for Sign Gesture Recognition Using Hybrid Deep Neural Architecture

The speech and hearing-impaired community use sign language as the primary means of communication. It is quite challenging for the general population to interpret or learn sign language completely. A sign language recognition system must be designed and developed to address this communication barrier. Most current sign language recognition systems rely on wearable sensors, keeping the recognition system unaffordable for most individuals. Moreover, the existing vision-based sign recognition frameworks do not consider all of the spatial and temporal information required for accurate recognition. A novel vison-based hybrid deep neural net methodology is proposed in this study for recognizing Indian and Russian sign gestures. The proposed framework is aimed to establish a single framework for tracking and extracting multi-semantic properties, such as non-manual components and manual co- articulations. Furthermore, spatial feature extraction from the sign gestures is deployed using a 3D deep neural net with atrous convolutions. The temporal and sequential feature extraction is carried out by employing attention-based Bi-LSTM. In addition, the distinguished abstract feature extraction is done using the modified autoencoders. The discriminative feature extraction for differentiating the sign gestures from unwanted transition gestures is done by leveraging the hybrid attention module. The experimentation of the proposed model has been carried out on the novel multi-signer Indo-Russian sign language dataset. The proposed sign language recognition framework with hybrid neural net yields better results than other state-of-the-art frameworks.

Intranasal Pressure Recording for Monitoring Mouse Respiration.

Respiration is a highly dynamic signal that influences voluntary behaviors including odor sampling and entrains rhythmic activity in the brain. Many techniques exist to record respiration with each exhibiting strengths and drawbacks given the ultimate goals of the respiration recording. Intranasal cannula implantation, coupled with pressure sensor recording, allows for temporal precision and detailed feature extraction of the respiratory waveform. Here we describe the implantation process and necessary recording equipment to effectively conduct intranasal pressure recording of respiration. This is an ideal method for understanding the dynamics of odor sampling in conjunction with olfactory sensory transmission.

Multimodal Spatiotemporal Feature Map for Dynamic Gesture Recognition

Gesture recognition technology enables machines to read human gestures and has significant application prospects in the fields of human-computer interaction and sign language translation. Existing researches usually use convolutional neural networks to extract features directly from raw gesture data for gesture recognition, but the networks are affected by much interference information in the input data and thus fit to some unimportant features. In this paper, we proposed a novel method for encoding spatio-temporal information, which can enhance the key features required for gesture recognition, such as shape, structure, contour, position and hand motion of gestures, thereby improving the accuracy of gesture recognition. This encoding method can encode arbitrarily multiple frames of gesture data into a single frame of the spatio-temporal feature map and use the spatio-temporal feature map as the input to the neural network. This can guide the model to fit important features while avoiding the use of complex recurrent network structures to extract temporal features. In addition, we designed two sub-networks and trained the model using a sub-network pre-training strategy that trains the sub-networks first and then the entire network, so as to avoid the sub-networks focusing too much on the information of a single category feature and being overly influenced by each other’s features. Experimental results on two public gesture datasets show that the proposed spatio-temporal information encoding method achieves advanced accuracy.

A Brain Network Analysis-Based Double Way Deep Neural Network for Emotion Recognition.

Constructing reliable and effective models to recognize human emotional states has become an important issue in recent years. In this article, we propose a double way deep residual neural network combined with brain network analysis, which enables the classification of multiple emotional states. To begin with, we transform the emotional EEG signals into five frequency bands by wavelet transform and construct brain networks by inter-channel correlation coefficients. These brain networks are then fed into a subsequent deep neural network block which contains several modules with residual connection and enhanced by channel attention mechanism and spatial attention mechanism. In the second way of the model, we feed the emotional EEG signals directly into another deep neural network block to extract temporal features. At the end of the two ways, the features are concatenated for classification. To verify the effectiveness of our proposed model, we carried out a series of experiments to collect emotional EEG from eight subjects. The average accuracy of the proposed model on our emotional dataset is 94.57%. In addition, the evaluation results on public databases SEED and SEED-IV are 94.55% and 78.91%, respectively, demonstrating the superiority of our model in emotion recognition tasks.

LWSleepNet: A lightweight attention-based deep learning model for sleep staging with singlechannel EEG.

Sleep is vital to human health, and sleep staging is an essential process in sleep assessment. However, manual classification is an inefficient task. Along with the increased demand for portable sleep quality detection devices, lightweight automatic sleep staging needs to be developed. This study proposes a novel attention-based lightweight deep learning model called LWSleepNet. A depthwise separable multi-resolution convolutional neural network is introduced to analyze the input feature map and captures features at multiple frequencies using two different sized convolutional kernels. The temporal feature extraction module divides the input into patches and feeds them into a multi-head attention block to extract time-dependent information from sleep recordings. The model's convolution operations are replaced with depthwise separable convolutions to minimize its number of parameters and computational cost. The model's performance on two public datasets (Sleep-EDF-20 and Sleep-EDF-78) was evaluated and compared with those of previous studies. Then, an ablation study and sensitivity analysis were performed to evaluate further each module. LWSleepNet achieves an accuracy of 86.6% and Macro-F1 score of 79.2% for the Sleep-EDF-20 dataset and an accuracy of 81.5% and Macro-F1 score of 74.3% for the Sleep-EDF-78 dataset with only 55.3 million floating-point operations per second and 180 K parameters. On two public datasets, LWSleepNet maintains excellent prediction performance while substantially reducing the number of parameters, demonstrating that our proposed Light multiresolution convolutional neural network and temporal feature extraction modules can provide excellent portability and accuracy and can be easily integrated into portable sleep monitoring devices.

EEG-Channel-Temporal-Spectral-Attention Correlation for Motor Imagery EEG Classification.

In brain-computer interface (BCI) work, how correctly identifying various features and their corresponding actions from complex Electroencephalography (EEG) signals is a challenging technology. However, most current methods do not consider EEG feature information in spatial, temporal and spectral domains, and the structure of these models cannot effectively extract discriminative features, resulting in limited classification performance. To address this issue, we propose a novel text motor-imagery EEG discrimination method, namely wavelet-based temporal-spectral-attention correlation coefficient (WTS-CC), to simultaneously consider the features and their weighting in spatial, EEG-channel, temporal and spectral domains in this study. The initial Temporal Feature Extraction (iTFE) module extracts the initial important temporal features of MI EEG signals. The Deep EEG-Channel-attention (DEC) module is then proposed to automatically adjust the weight of each EEG channel according to its importance, thereby effectively enhancing more important EEG channels and suppressing less important EEG channels. Next, the Wavelet-based Temporal-Spectral-attention (WTS) module is proposed to obtain more significant discriminative features between different MI tasks by weighting features on two-dimensional time-frequency maps. Finally, a simple discrimination module is used for MI EEG discrimination. The experimental results indicate that the proposed text WTS-CC method can achieve promising discrimination performance that outperforms the state-of-the-art methods in terms of classification accuracy, Kappa coefficient, F1 score, and AUC on three public datasets.

FoleyGAN: Visually Guided Generative Adversarial Network-Based Synchronous Sound Generation in Silent Videos

Deep learning based visual-to-sound generation systems have been developed that identify and create audio features from video signals. However, these techniques often fail to consider the time-synchronicity of the visual and audio features. In this paper we introduce a novel method for guiding a class-conditioned GAN to synthesize representative audio with temporally extracted visual information. We accomplish this visual-to-sound generation task by adapting the synchronicity traits between the audio-visual modalities. Our proposed FoleyGAN model is capable of conditioning action sequences of visual events leading to the generation of visually aligned realistic soundtracks. We expanded our previously proposed Automatic Foley data set. We evaluated FoleyGAN's synthesized sound output through human surveys that show noteworthy (on average 81%) audio-visual synchronicity performance. Our approach outperforms other baseline models and audio-visual data sets in statistical and ablation experiments achieving improved IS, FID and NDB scores. In ablation analysis we showed the significance of our visual and temporal feature extraction method as well as augmented performance of our generation network. Overall, our FoleyGAN model showed sound retrieval accuracy of 76.08% surpassing existing visual-to-audio synthesis deep neural networks.

Forgery detection of low quality deepfake videos

The rapid growth of online media over different social media platforms or over the internet along with many benefits have some negative effects as well. Deep learning has many positive applications like medical, animations and cybersecurity etc. But over the past few years, it is observed that it is been used for negative aspects as well such as defaming, black-mailing and creating privacy concerns for the general public. Deepfake is common terminology used for facial forgery of a person in media like images or videos.The advancement in the forgery creation area have challenged the researchers to create and develop advance forgery detection systems capable to detect facial forgeries. Proposed forgery detection system works on the CNN-LSTM model in which we first extracted faces from the frames using MTCNN then performed spatial feature extraction using pretrained Xception network and then used LSTM for temporal feature extraction. At the end classification is performed to predict the video as real or fake. The system is capable to detect low quality videos. The current system has shown good accuracy results for detecting real or fake videos on the Google deepfake AI dataset.

STC-NLSTMNet: An Improved Human Activity Recognition Method Using Convolutional Neural Network with NLSTM from WiFi CSI

Human activity recognition (HAR) has emerged as a significant area of research due to its numerous possible applications, including ambient assisted living, healthcare, abnormal behaviour detection, etc. Recently, HAR using WiFi channel state information (CSI) has become a predominant and unique approach in indoor environments compared to others (i.e., sensor and vision) due to its privacy-preserving qualities, thereby eliminating the need to carry additional devices and providing flexibility of capture motions in both line-of-sight (LOS) and non-line-of-sight (NLOS) settings. Existing deep learning (DL)-based HAR approaches usually extract either temporal or spatial features and lack adequate means to integrate and utilize the two simultaneously, making it challenging to recognize different activities accurately. Motivated by this, we propose a novel DL-based model named spatio-temporal convolution with nested long short-term memory (STC-NLSTMNet), with the ability to extract spatial and temporal features concurrently and automatically recognize human activity with very high accuracy. The proposed STC-NLSTMNet model is mainly comprised of depthwise separable convolution (DS-Conv) blocks, feature attention module (FAM) and NLSTM. The DS-Conv blocks extract the spatial features from the CSI signal and add feature attention modules (FAM) to draw attention to the most essential features. These robust features are fed into NLSTM as inputs to explore the hidden intrinsic temporal features in CSI signals. The proposed STC-NLSTMNet model is evaluated using two publicly available datasets: Multi-environment and StanWiFi. The experimental results revealed that the STC-NLSTMNet model achieved activity recognition accuracies of 98.20% and 99.88% on Multi-environment and StanWiFi datasets, respectively. Its activity recognition performance is also compared with other existing approaches and our proposed STC-NLSTMNet model significantly improves the activity recognition accuracies by 4% and 1.88%, respectively, compared to the best existing method.

Nystagmus patterns classification framework based on deep learning and optical flow

Benign paroxysmal positional vertigo (BPPV) is the most common vestibular peripheral vertigo disease characterized by brief recurrent vertigo with positional nystagmus. Clinically, it is common to recognize the patterns of nystagmus by analyzing infrared nystagmus videos of patients. However, the existing approaches cannot effectively recognize different patterns of nystagmus, especially the torsional nystagmus. To improve the performance of recognizing different nystagmus patterns, this paper contributes an automatic recognizing method of BPPV nystagmus patterns based on deep learning and optical flow to assist doctors in analyzing the types of BPPV. Firstly, we present an adaptive method for eliminating invalid frames that caused by eyelid occlusion or blinking in nystagmus videos and an adaptive method for segmenting the iris and pupil area from video frames quickly and efficiently. Then, we use a deep learning-based optical flow method to extract nystagmus information. Finally, we propose a nystagmus video classification network (NVCN) to categorize the patterns of nystagmus. We use ConvNeXt to extract eye movement features and then use LSTM to extract temporal features. Experiments conducted on the clinically collected datasets of infrared nystagmus videos show that the NVCN model achieves an accuracy of 94.91% and an F1 score of 93.70% on nystagmus patterns classification task as well as an accuracy of 97.75% and an F1 score of 97.48% on torsional nystagmus recognition task. The experimental results prove that the framework we propose can effectively recognize different patterns of nystagmus.

Soft-Sensor Development through Deep Learning with Spatial and Temporal Feature Extraction of Complex Processes

Data-driven soft sensing techniques have been widely applied to predict the quality variables in the process industry. The temporal and spatial structure information is important for process modeling because it helps extract the nonlinear and dynamic characteristics from data. Deep learning can be used to mimic the physical linking structure by designing similar node connections in the neural network. Taking a complex process with countercurrent unit connection as a demonstration, this paper proposes using a bidirectional long short-term memory-based network with spatiotemporal feature extraction for soft-sensor development. An attention mechanism is further conducted for the feature learning from space and time simultaneously. The effectiveness of the proposed method is validated through a simulation example and a real industrial example.