Improve Recognition Performance Research Articles

Multimodal Sparse Transformer Network for Audio-Visual Speech Recognition.

Automatic speech recognition (ASR) is the major human-machine interface in many intelligent systems, such as intelligent homes, autonomous driving, and servant robots. However, its performance usually significantly deteriorates in the presence of external noise, leading to limitations of its application scenes. The audio-visual speech recognition (AVSR) takes visual information as a complementary modality to enhance the performance of audio speech recognition effectively, particularly in noisy conditions. Recently, the transformer-based architectures have been used to model the audio and video sequences for the AVSR, which achieves a superior performance. However, its performance may be degraded in these architectures due to extracting irrelevant information while modeling long-term dependences. In addition, the motion feature is essential for capturing the spatio-temporal information within the lip region to best utilize visual sequences but has not been considered in the AVSR tasks. Therefore, we propose a multimodal sparse transformer network (MMST) in this article. The sparse self-attention mechanism can improve the concentration of attention on global information by selecting the most relevant parts wisely. Moreover, the motion features are seamlessly introduced into the MMST model. We subtly allow motion-modality information to flow into visual modality through the cross-modal attention module to enhance visual features, thereby further improving recognition performance. Extensive experiments conducted on different datasets validate that our proposed method outperforms several state-of-the-art methods in terms of the word error rate (WER).

SAR-ATR with knowledge hierarchy division and information dissemination networks

Synthetic aperture radar (SAR) images possess diverse domain characteristics that describe targets from multiple perspectives, including scattering center, intensity, phase, azimuth angle, and shadow. The imaging mechanism of SAR images significantly differs from optical images due to its use of radio waves instead of visible light for imaging purposes. However, current deep learning (DL)-based automatic target recognition (ATR) approaches for SAR primarily focus on the intensity information while neglecting the comprehensive consideration of SAR domain characteristics. This limitation results in insufficient expression of target features and hampers further improvement in SAR target recognition performance. To address these issues, this paper proposes a novel SAR target recognition method called KDNets based on information dissemination networks (IDNets) and knowledge hierarchy division (KnHD). KDNets effectively enhances the feature representation ability of SAR targets by exploiting multiple domain characteristics at sample, feature, and decision levels. Specifically, IDNets incorporates scattering center and azimuth angle at sample and feature levels to extract rich multi-scale semantic features for representing SAR targets. At the decision stage, KnHD utilizes azimuth information to rapidly search and fuse prior knowledge about targets from a knowledge base to achieve interpretable and high-precision SAR target recognition. Experimental results on the MSTAR dataset and MA-SAR dataset demonstrate that our proposed KDNets can achieve state-of-the-art performance in SAR target recognition tasks, with the accuracy of 99.75 % in the MSTAR dataset and 98.61 % in the MA-SAR dataset, validating its effectiveness and superiority. Furthermore, this study highlights the significant potential and application prospects of leveraging domain characteristics in SAR-ATR tasks.

ReFuSeAct: Representation fusion using self-supervised learning for activity recognition in next generation networks

Over the years, wearable sensors have gained a lot of attention from the research community due to their non-invasive nature, adoption of sensors by general public, and their applicability in healthcare services. With the advancements in communication networks, machine learning methods, and wearable sensor deployment, it is essential to design a method that could accurately classify human activities while reducing the dependence on annotated data. Traditional machine learning approaches require large-scale annotated data in order to provide a reasonable recognition performance. Recently, self-supervised learning methods are proposed but they are either limited to single sensor devices or fail to model intra-modal correlations within the self-supervised learning paradigm. In this work, we propose Representation Fusion using Self-supervised learning for Activity Recognition (ReFuSeAct) framework that uses modality-specific encoders, attention encoders, and decision-level fusion strategies to address the aforementioned limitations. The self-supervised learning paradigm ensures that the method achieves better performance even with less amount of annotated data. The architecture proposed for modality-specific encoder ensures that extraction of representative features that could help in improving recognition performance. The feature-level fusion performed using the proposed attention encoders enhances the quality of representative features that could be used in supervised learning phase. Finally, the decision-level fusion strategy enhances the activity recognition accuracy in comparison to the single deep learning classifier. Our experimental analysis shows that the proposed approach records 9.1% improvement over semi-supervised learning baselines and more than 2% improvement in comparison to existing self-supervised learning approaches.

Hybrid Deep Neural Network Framework Combining Skeleton and Gait Features for Pathological Gait Recognition.

Human skeleton data obtained using a depth camera have been used for pathological gait recognition to support doctor or physician diagnosis decisions. Most studies for skeleton-based pathological gait recognition have used either raw skeleton sequences directly or gait features, such as gait parameters and joint angles, extracted from raw skeleton sequences. We hypothesize that using skeleton, joint angles, and gait parameters together can improve recognition performance. This study aims to develop a deep neural network model that effectively combines different types of input data. We propose a hybrid deep neural network framework composed of a graph convolutional network, recurrent neural network, and artificial neural network to effectively encode skeleton sequences, joint angle sequences, and gait parameters, respectively. The features extracted from three different input data types are fused and fed into the final classification layer. We evaluate the proposed model on two different skeleton datasets (a simulated pathological gait dataset and a vestibular disorder gait dataset) that were collected using an Azure Kinect. The proposed model, with multiple types of input, improved the pathological gait recognition performance compared to single input models on both datasets. Furthermore, it achieved the best performance among the state-of-the-art models for skeleton-based action recognition.

Toward enriched decoding of mandarin spontaneous speech

A deep neural network (DNN)-based automatic speech recognition (ASR) method for enriched decoding of Mandarin spontaneous speech is proposed. It adopts an enhanced approach over the baseline model built with factored time delay neural networks (TDNN-f) and rescored with RNNLM to first building a baseline system composed of a TDNN-f acoustic model (AM), a trigram language model (LM), and a recurrent neural network language model (RNNLM) to generate a word lattice. It then sequentially incorporates a multi-task Part-of-Speech-RNNLM (POS-RNNLM), a hierarchical prosodic model (HPM), and a reduplication-word LM (RLM) into the decoding process by expanding the word lattice and performing rescoring to improve recognition performance and enrich the decoding output with syntactic parameters of POS and punctuation (PM), prosodic tags of word-juncture break types and syllable prosodic states, and an edited recognition text with reduplication words being eliminated. Experimental results on the Mandarin conversational dialogue corpus (MCDC) showed that SER, CER, and WER of 13.2 %, 13.9 %, and 19.1 % were achieved when incorporating the POS-RNNLM and HPM into the baseline system. They represented relative SER, CER, and WER reductions of 7.7 %, 7.9 % and 5.0 % as comparing with those of the baseline system. Futhermore, the use of the RLM resulted in additional 3 %, 4.6 %, and 4.5 % relative SER, CER, and WER reductions through eliminating reduplication words.

Visual modalities-based multimodal fusion for surgical phase recognition

Surgical workflow analysis is essential to help optimize surgery by encouraging efficient communication and the use of resources. However, the performance of phase recognition is limited by the use of information related to the presence of surgical instruments. To address the problem, we propose visual modality-based multimodal fusion for surgical phase recognition to overcome the limited diversity of information such as the presence of instruments. Using the proposed methods, we extracted a visual kinematics-based index related to using instruments, such as movement and their interrelations during surgery. In addition, we improved recognition performance using an effective convolutional neural network (CNN)-based fusion method for visual features and a visual kinematics-based index (VKI). The visual kinematics-based index improves the understanding of a surgical procedure since information is related to instrument interaction. Furthermore, these indices can be extracted in any environment, such as laparoscopic surgery, and help obtain complementary information for system kinematics log errors. The proposed methodology was applied to two multimodal datasets, a virtual reality (VR) simulator-based dataset (PETRAW) and a private distal gastrectomy surgery dataset, to verify that it can help improve recognition performance in clinical environments. We also explored the influence of a visual kinematics-based index to recognize each surgical workflow by the instrument’s existence and the instrument’s trajectory. Through the experimental results of a distal gastrectomy video dataset, we validated the effectiveness of our proposed fusion approach in surgical phase recognition. The relatively simple yet index-incorporated fusion we propose can yield significant performance improvements over only CNN-based training and exhibits effective training results compared to fusion based on Transformers, which require a large amount of pre-trained data.

Does Tactile Stimulation of the Face Affect the Processing of Other Faces? Neural and Behavioural Effects of Facial Touch

ABSTRACTThe integration of vision and touch is proposed as a critical factor for processing one’s own body and the bodies of others in the brain. We hypothesize that tactile stimulation on an individual’s face may change the ability to process the faces of other, but not the processing of other visual images. We aimed to determine if facial touch increased the activity of the mirror system and face recognition memory of the observer. Therefore, mu suppression was measured to compare the effect of facial touch in performing two visual tasks. The participants observed faces and non-face visual images under two sets of conditions. In the first condition, a robotic finger touched the participant’s cheek while in the second condition, no touch occurred. Upon each observational task, the participants were given in a recognition test. Behavioral results indicated that facial touch improved recognition performance for faces, but not for non-face visual images. Tactile stimulation increased mu suppression in both occipital and central electrodes during face processing; however, the suppression did not significantly change during non-face visual processing. Our findings support the concept that the brain uses a self-body representation, as a reference to understand the mental states or behaviors of others.

Research on Weibo New Word Recognition based on Weibo Data and Statistical Information

One of the key challenges in the field of Chinese information processing is the recognition of Weibo new words, which has a profound impact on machine translation and text classification. As Weibo has become the most used social platform for internet users, mining new vocabulary from Weibo data not only helps to deeply understand the data itself, but also provides personalized recommendation services for users. Although a large amount of research has focused on the recognition of Weibo new words, specialized research in this field is still scarce. In this article, we propose a Weibo new word recognition strategy that combines Weibo content features and statistical information. Firstly, extract repetitive vocabulary from Weibo topic names, and then use various methods such as absolute frequency, relative frequency, mutual information, and information entropy to filter for incorrect vocabulary. The experimental results show that by setting appropriate thresholds, incorrect vocabulary can be effectively filtered out, thereby improving recognition performance.

Intelligent vision for the detection of chemistry glassware toward AI robotic chemists

One of the key steps to make an artificially intelligent (AI) and robotic chemist is the introduction of machine vision for guiding the experiment operation in the AI-redefined laboratory. In order to realize the targets, the prerequisites are to innovate/implement the intelligent vision for the detection of chemistry glassware. Here, we reported a computer vision method based on You only look once (YOLO) with a self-developed Chemical Vessel Identification Dataset (CViG) for the improvement of classification and recognition performance. The training dataset has been collected that includes 4072 images in real-time chemical laboratory. Three models, YOLOv5s, Slim-YOLOv5s and YOLOv7, have been exploited for the recognition of seven types of glassware in the condition of different scenarios (recognition distance, light and dark, stationary and moving). The improved Slim-YOLOv5s exhibited better recognition ability in various scenes, and the recognition accuracy of chemical vessels is improved by 1.51 % compared with YOLOv5s, and the size of the model is reduced from 14.4 MB to 11.0 MB. Slim-YOLOv5s's mAP is similar to YOLOv7's ability with a disadvantage of large volume, suggested that the improved Slim-YOLOv5s clearly has more advantages in terms of embedded requirements. This vision-assisted system capable of classifying chemical containers accurately in the scenarios of real-time chemical experiments will provide a good vision solution in the frontier fields of automated machine chemistry.

Handwritten Javanese script recognition method based 12-layers deep convolutional neural network and data augmentation

Although numerous studies have been conducted on handwritten recognition, there is little and non-optimal research on Javanese script recognition due to its limitation to basic characters. Therefore, this research proposes the design of a handwritten Javanese Script recognition method based on twelve layers deep convolutional neural network (DCNN), consisting of four convolutions, two pooling, and five fully connected (FC) layers, with SoftMax classifiers. Five FC layers were proposed in this research to conduct the learning process in stages to achieve better learning outcomes. Due to the limited number of images in the Javanese script dataset, an augmentation process is needed to improve recognition performance. This method obtained 99.65% accuracy using seven types of geometric augmentation and the proposed DCNN model for 120 Javanese script character classes. It consists of 20 basic characters plus 100 others from the compound of basic and vowels characters.

CRABR-Net: A Contextual Relational Attention-Based Recognition Network for Remote Sensing Scene Objective.

Remote sensing scene objective recognition (RSSOR) plays a serious application value in both military and civilian fields. Convolutional neural networks (CNNs) have greatly enhanced the improvement of intelligent objective recognition technology for remote sensing scenes, but most of the methods using CNN for high-resolution RSSOR either use only the feature map of the last layer or directly fuse the feature maps from various layers in the "summation" way, which not only ignores the favorable relationship information between adjacent layers but also leads to redundancy and loss of feature map, which hinders the improvement of recognition accuracy. In this study, a contextual, relational attention-based recognition network (CRABR-Net) was presented, which extracts different convolutional feature maps from CNN, focuses important feature content by using a simple, parameter-free attention module (SimAM), fuses the adjacent feature maps by using the complementary relationship feature map calculation, improves the feature learning ability by using the enhanced relationship feature map calculation, and finally uses the concatenated feature maps from different layers for RSSOR. Experimental results show that CRABR-Net exploits the relationship between the different CNN layers to improve recognition performance, achieves better results compared to several state-of-the-art algorithms, and the average accuracy on AID, UC-Merced, and RSSCN7 can be up to 96.46%, 99.20%, and 95.43% with generic training ratios.

Cross-modal alignment and translation for missing modality action recognition

Multimodal data provide complementary information on the same context, leading to performance improvement in video action recognition. However, in reality, not all modalities are available at test time. To this end, we propose Cross-Modal Alignment and Translation (CMAT) framework for action recognition that is robust to missing modalities. Specifically, our framework first aligns representations of multiple modalities from the same video sample through contrastive learning, effectively alleviating the bias with respect to the type of missing modality. Then, CMAT learns to translate representations of one modality into that of another modality. This allows the representations of the missing modalities to be generated from the remaining modalities during the testing. Accordingly, CMAT fully utilizes multimodal information obtained through abundant interactions across modalities. The proposed CMAT achieves state-of-the-art performances in both complete and missing modality settings on NTU RGB+D, NTU RGB+D 120, and Northwestern-UCLA datasets. Moreover, extensive ablation studies demonstrate the effectiveness of our design.

Deep cross-domain transfer for emotion recognition via joint learning

Deep learning has been applied to achieve significant progress in emotion recognition from multimedia data. Despite such substantial progress, existing approaches are hindered by insufficient training data, leading to weak generalisation under mismatched conditions. To address these challenges, we propose a learning strategy which jointly transfers emotional knowledge learnt from rich datasets to source-poor datasets. Our method is also able to learn cross-domain features, leading to improved recognition performance. To demonstrate the robustness of the proposed learning strategy, we conducted extensive experiments on several benchmark datasets including eNTERFACE, SAVEE, EMODB, and RAVDESS. Experimental results show that the proposed method surpassed existing transfer learning schemes by a significant margin.

Optimized Classifier Learning for Face Recognition Performance Boost in Security and Surveillance Applications.

Face recognition has become an integral part of modern security processes. This paper introduces an optimization approach for the quantile interval method (QIM), a promising classifier learning technique used in face recognition to create face templates and improve recognition accuracy. Our research offers a three-fold contribution to the field. Firstly, (i) we strengthened the evidence that QIM outperforms other contemporary template creation approaches. For this reason, we investigate seven template creation methods, which include four cluster description-based methods and three estimation-based methods. Further, (ii) we extended testing; we use a nearly four times larger database compared to the previous study, which includes a new set, and we report the recognition performance on this extended database. Additionally, we distinguish between open- and closed-set identification. Thirdly, (iii) we perform an evaluation of the cluster estimation-based method (specifically QIM) with an in-depth analysis of its parameter setup in order to make its implementation feasible. We provide instructions and recommendations for the correct parameter setup. Our research confirms that QIM's application in template creation improves recognition performance. In the case of automatic application and optimization of QIM parameters, improvement recognition is about 4-10% depending on the dataset. In the case of a too general dataset, QIM also provides an improvement, but the incorporation of QIM into an automated algorithm is not possible, since QIM, in this case, requires manual setting of optimal parameters. This research contributes to the advancement of secure and accurate face recognition systems, paving the way for its adoption in various security applications.

Person Re-Identification Method Based on Dual Descriptor Feature Enhancement.

Person re-identification is a technology used to identify individuals across different cameras. Existing methods involve extracting features from an input image and using a single feature for matching. However, these features often provide a biased description of the person. To address this limitation, this paper introduces a new method called the Dual Descriptor Feature Enhancement (DDFE) network, which aims to emulate the multi-perspective observation abilities of humans. The DDFE network uses two independent sub-networks to extract descriptors from the same person image. These descriptors are subsequently combined to create a comprehensive multi-view representation, resulting in a significant improvement in recognition performance. To further enhance the discriminative capability of the DDFE network, a carefully designed training strategy is employed. Firstly, the CurricularFace loss is introduced to enhance the recognition accuracy of each sub-network. Secondly, the DropPath operation is incorporated to introduce randomness during sub-network training, promoting difference between the descriptors. Additionally, an Integration Training Module (ITM) is devised to enhance the discriminability of the integrated features. Extensive experiments are conducted on the Market1501 and MSMT17 datasets. On the Market1501 dataset, the DDFE network achieves an mAP of 91.6% and a Rank1 of 96.1%; on the MSMT17 dataset, the network achieves an mAP of 69.9% and a Rank1 of 87.5%. These outcomes outperform most SOTA methods, highlighting the significant advancement and effectiveness of the DDFE network.

Emotion recognition and mood along the menstrual cycle

Previous studies have demonstrated menstrual cycle dependent changes in the recognition of facial emotional expressions, specifically the expression of fear, anger, sadness or disgust. While some studies demonstrate an improvement of emotion recognition performance during the peri-ovulatory phase, when estradiol levels peak, other studies demonstrate a deterioration of emotion recognition performance during the mid-luteal phase, when progesterone levels peak. It has been hypothesized, that these changes in emotion recognition performance mirror mood changes along the menstrual cycle. In the present study, we investigate, whether changes in emotion recognition performance along the menstrual cycle are mediated by mood changes along the menstrual cycle. In a combined cross-sectional and longitudinal study design, two large samples of women completed an emotion recognition task, as well as several mood questionnaires during their menses, peri-ovulatory or mid-luteal cycle phase. 65 women completed the task thrice, once during each cycle phase, order counterbalanced. In order to control for potential learning effects, a sample of 153 women completed the task only once in one of the three cycle phases. In both samples, results demonstrated no significant changes in emotion recognition performance along the menstrual cycle, irrespective of the performance measure investigated (accuracy, reaction time, frequency of emotion classifications) and irrespective of the emotion displayed. Bayesian statistics provided very strong evidence for the null hypothesis, that emotion recognition does not change along the menstrual cycle. There was also no moderation of emotion recognition changes along the menstrual cycle by mood changes along the menstrual cycle. Mood changes along the menstrual cycle followed the expected pattern with highest positive affect and least premenstrual symptoms around ovulation and lowest positive affect, but strongest premenstrual symptoms during menses. Interestingly, premenstrual symptoms were negatively related to estradiol, suggesting a protective effect of estrogen during the luteal cycle phase against mood worsening during the premenstrual phase.

Few-shot person re-identification based on Feature Set Augmentation and Metric Fusion

Person re-identification identifies pedestrians by analyzing image information from surveillance videos. However, it faces challenges like occlusion, changing lighting, and costly annotation. Thus, it is often performed in a few-shot environment with limited images. In response to the problem of insufficient available pedestrian images in person re-identification, a few-shot person re-identification method based on feature set augmentation and metric fusion is proposed. In this work, firstly, a feature augmentation method is introduced into the feature embedding module. This method introduces multi-head self-attention in different feature extraction layers and spatial attention in feature fusion of different feature extraction layers, which can extract more diverse and discriminative pedestrian features. Secondly, a dual metric method combining Euclidean and cosine distance is proposed in the metric module to comprehensively measure the absolute spatial distance and directional difference of pedestrian features. In this way, the reliability of pedestrian similarity measurement is improved. Then, pedestrian feature similarity scores are obtained separately using the dual metric and relation metric methods. Finally, the combined metric score is obtained by weighted fusion, and the combined metric score is used to construct the joint loss to realize the overall optimization and training of the network. Experimental results on three small datasets, Market-mini, Duke-mini, and MSMT17-mini, show that the proposed method significantly improves recognition performance compared to other few-shot learning algorithms. Specifically, in scenarios 5-way 1-shot and 5-way 5-shot, the average recognition accuracies are 92.54% and 96.99%, 87.93% and 96.08%, and 71.68% and 84.51%, respectively.

Mean Teacher-Based Cross-Domain Activity Recognition Using WiFi Signals

WiFi Channel State Information (CSI)-based activity recognition has initiated a great many studies because of wide availability and privacy protection. However, general recognition approaches still struggle to generalize beyond the source domain of training data, i.e., well-trained models might not be suitable to target data with unseen subjects or environments. Existing solutions, such as few-shot learning-based and data augmentation-based approaches, either require a few labeled target samples which is difficult to be collected especially for old target users, or inappropriately treat augmented samples with different amounts of noise. To overcome these limitations, we propose a Mean Teacher-based cross-domain human activity recognition framework using WiFi CSI, WiTeacher. In this framework, to address shift between source and target domains, we built a label smoothing-based classification loss, where the input data are the target-like samples generated by StyleGAN, and corresponding label values are dynamically adjusted by our designed adaptive label smoothing method. To enhance the model robustness, we devise a sample relation-based consistency regularization term to keep the distances of the two samples with and without perturbations invariant, which can exploit the relationships between samples to improve recognition performance. The experiments illustrate that WiTeacher achieves obvious gains without requiring any annotation data from the target domain.

Space-and-speaker-aware acoustic modeling with effective data augmentation for recognition of multi-array conversational speech

We propose a space-and-speaker-aware (SSA) approach to acoustic modeling (AM), denoted as SSA-AM, to improve system performances of automatic speech recognition (ASR) in distant multi-array conversational scenarios. In contrast to conventional AM which only uses spectral features from a target speaker as inputs, the inputs to SSA-AM consists of speech features from both the target and interfering speakers, which contain discriminative information from different speakers, including spatial information embedded in interaural phase differences (IPDs) between individual interfering speakers and the target speaker. In the proposed SSA-AM framework, we explore four acoustic model architectures consisting of different combinations of four neural networks, namely deep residual network, factorized time delay neural network, self-attention and residual bidirectional long short-term memory neural network. Various data augmentation techniques are adopted to expand the training data to include different options of beamformed speech obtained from multi-channel speech enhancement. Evaluated on the recent CHiME-6 Challenge Track 1, our proposed SSA-AM framework achieves consistent recognition performance improvements when compared with the official baseline acoustic models. Furthermore, SSA-AM outperforms acoustic models without explicitly using the space and speaker information. Finally, our data augmentation schemes are shown to be especially effective for compact model designs. Code is released at https://github.com/coalboss/SSA_AM.

Contrastive self-supervised representation learning without negative samples for multimodal human action recognition.

Action recognition is an important component of human-computer interaction, and multimodal feature representation and learning methods can be used to improve recognition performance due to the interrelation and complementarity between different modalities. However, due to the lack of large-scale labeled samples, the performance of existing ConvNets-based methods are severely constrained. In this paper, a novel and effective multi-modal feature representation and contrastive self-supervised learning framework is proposed to improve the action recognition performance of models and the generalization ability of application scenarios. The proposed recognition framework employs weight sharing between two branches and does not require negative samples, which could effectively learn useful feature representations by using multimodal unlabeled data, e.g., skeleton sequence and inertial measurement unit signal (IMU). The extensive experiments are conducted on two benchmarks: UTD-MHAD and MMAct, and the results show that our proposed recognition framework outperforms both unimodal and multimodal baselines in action retrieval, semi-supervised learning, and zero-shot learning scenarios.