Connectionist Temporal Classification Research Articles

Spatial Transform Depthwise Over-Parameterized Convolution Recurrent Neural Network for License Plate Recognition in Complex Environment

Abstract Automatic license plate recognition (ALPR) system has been widely used in intelligent transportation and other fields. However, in complex environments such as vehicle sound source localization, poor illumination, or bad weather conditions, ALPR is still a challenging problem. Aiming at the problem, an end-to-end deep learning framework is developed based on depthwise over-parameterized convolution recurrent neural network for license plate character recognition. The proposed framework is composed as follows: (i) license plate correcting module based on spatial transformation network; (ii) feature extraction module based on depthwise over-parameterized convolution; (iii) sequence annotation module based on bidirectional long short-term memory; and (iv) regularized sequence decoding module based on connectionist temporal classification with maximum conditional entropy. Two open-source datasets of Chinese License Plate Datasets (SYSU) and Chinese City Parking Dataset (CCPD) are used to verify the performance of the algorithm. The proposed end-to-end framework can effectively rectify distorted and inclined license plates in spatial domain. It can recognize license plates without complex character segmentation process. Compared with some current state-of-art algorithms, the proposed algorithm achieved the best performance with the recognition accuracy of 96.31% and 88.31% based on the two datasets of SYSU and CCPD, respectively.

Benchmarking online sequence-to-sequence and character-based handwriting recognition from IMU-enhanced pens

Handwriting is one of the most frequently occurring patterns in everyday life and with it comes challenging applications such as handwriting recognition, writer identification and signature verification. In contrast to offline HWR that only uses spatial information (i.e., images), online HWR uses richer spatio-temporal information (i.e., trajectory data or inertial data). While there exist many offline HWR datasets, there are only little data available for the development of OnHWR methods on paper as it requires hardware-integrated pens. This paper presents data and benchmark models for real-time sequence-to-sequence learning and single character-based recognition. Our data are recorded by a sensor-enhanced ballpoint pen, yielding sensor data streams from triaxial accelerometers, a gyroscope, a magnetometer and a force sensor at 100 Hz. We propose a variety of datasets including equations and words for both the writer-dependent and writer-independent tasks. Our datasets allow a comparison between classical OnHWR on tablets and on paper with sensor-enhanced pens. We provide an evaluation benchmark for seq2seq and single character-based HWR using recurrent and temporal convolutional networks and transformers combined with a connectionist temporal classification (CTC) loss and cross-entropy (CE) losses. Our convolutional network combined with BiLSTMs outperforms transformer-based architectures, is on par with InceptionTime for sequence-based classification tasks and yields better results compared to 28 state-of-the-art techniques. Time-series augmentation methods improve the sequence-based task, and we show that CE variants can improve the single classification task. Our implementations together with the large benchmark of state-of-the-art techniques of novel OnHWR datasets serve as a baseline for future research in the area of OnHWR on paper.

PageNet: Towards End-to-End Weakly Supervised Page-Level Handwritten Chinese Text Recognition

Handwritten Chinese text recognition (HCTR) has been an active research topic for decades. However, most previous studies solely focus on the recognition of cropped text line images, ignoring the error caused by text line detection in real-world applications. Although some approaches aimed at page-level text recognition have been proposed in recent years, they either are limited to simple layouts or require very detailed annotations including expensive line-level and even character-level bounding boxes. To this end, we propose PageNet for end-to-end weakly supervised page-level HCTR. PageNet detects and recognizes characters and predicts the reading order between them, which is more robust and flexible when dealing with complex layouts including multi-directional and curved text lines. Utilizing the proposed weakly supervised learning framework, PageNet requires only transcripts to be annotated for real data; however, it can still output detection and recognition results at both the character and line levels, avoiding the labor and cost of labeling bounding boxes of characters and text lines. Extensive experiments conducted on five datasets demonstrate the superiority of PageNet over existing weakly supervised and fully supervised page-level methods. These experimental results may spark further research beyond the realms of existing methods based on connectionist temporal classification or attention. The source code is available at https://github.com/shannanyinxiang/PageNet .

Speech GAU: A Single Head Attention for Mandarin Speech Recognition for Air Traffic Control

The rise of end-to-end (E2E) speech recognition technology in recent years has overturned the design pattern of cascading multiple subtasks in classical speech recognition and achieved direct mapping of speech input signals to text labels. In this study, a new E2E framework, ResNet–GAU–CTC, is proposed to implement Mandarin speech recognition for air traffic control (ATC). A deep residual network (ResNet) utilizes the translation invariance and local correlation of a convolutional neural network (CNN) to extract the time-frequency domain information of speech signals. A gated attention unit (GAU) utilizes a gated single-head attention mechanism to better capture the long-range dependencies of sequences, thus attaining a larger receptive field and contextual information, as well as a faster training convergence rate. The connectionist temporal classification (CTC) criterion eliminates the need for forced frame-level alignments. To address the problems of scarce data resources and unique pronunciation norms and contexts in the ATC field, transfer learning and data augmentation techniques were applied to enhance the robustness of the network and improve the generalization ability of the model. The character error rate (CER) of our model was 11.1% on the expanded Aishell corpus, and it decreased to 8.0% on the ATC corpus.

Pho(SC)-CTC—a hybrid approach towards zero-shot word image recognition

Annotating words in a historical document image archive for word image recognition purpose demands time and skilled human resource (like historians, paleographers). In a real-life scenario, obtaining sample images for all possible words is also not feasible. However, zero-shot learning methods could aptly be used to recognize unseen/out-of-lexicon words in such historical document images. Based on previous state-of-the-art method for zero-shot word recognition “Pho(SC)Net”, we propose a hybrid model based on the CTC framework (Pho(SC)-CTC) that takes advantage of the rich features learned by Pho(SC)Net followed by a “connectionist temporal classification” (CTC) framework to perform the final classification. Encouraging results were obtained on two publicly available historical document datasets and one synthetic handwritten dataset, which justifies the efficacy of Pho(SC)-CTC and Pho(SC)Net.

Spatial attention contrastive network for scene text recognition

At present, most scene text recognition methods achieve good performance by training models on many synthetic data. However, many data lead to huge storage space and large amount of calculation. And there is a gap between synthetic and real data. To solve these problems, we use a few real data to train a novel proposed model named spatial attention contrastive network (SAC-Net). The SAC-Net consists of a background suppression network (BSNet), a feature encoder, an attention decoder (ADEer), and a feature contrastive network (FCNet). The BSNet based on U-Net is used to reduce the interference of background. For relatively low prediction accuracy brought by connectionist temporal classification, we design an ADEer to improve performance by using convolutional attention mechanism. Based on data augmentation, we design a FCNet which belongs to contrastive learning. Finally, our SAC-Net is almost equivalent to the state-of-the-art model trained on a few real data for word accuracy on six benchmark test datasets.

Intelligent Deep Learning Empowered Text Detection Model from Natural Scene Images

The scene Text Recognition process has become a hot research topic and a challenging task owing to the complicated background, varying light intensities, colors, font styles, and sizes. Text extraction from natural scene images encompasses two main processes: text detection and text recognition. The latest advancements in Machine Learning (ML) and Deep Learning (DL) concepts can effectually automate the text detection and recognition process by training the model properly. In this view, this paper presents an Automated DL empowered Text Detection model from Natural Scene Images (ADLTD-NSI). The ADLTD-NSI technique includes two important processes: text detection and text recognition. Firstly, a single shot detector (SSD) with Inception-v2 as a baseline model is employed for text detection, an object detector based on the VGG-16 framework for feature map extraction followed by six convolution layers. Secondly, Convolutional Recurrent Neural Network (CRNN) technique is utilized for the text recognition process. Besides, the recurrent layers in the CRNN model utilize long short-term memory (LSTM) for encoding the sequence of feature vectors. Lastly, Connectionist Temporal Classification (CTC) loss is applied to predict text labels equivalent to the sequences from the recurrent layers. A wide range of experiments was carried out on benchmark COCO datasets, and the results are examined in several aspects. The experimental outcomes showcased the better performance of the ADLTD-NSI technique over the other compared methods with a maximum accuracy of 96.78%.

MSRCall: a multi-scale deep neural network to basecall Oxford Nanopore sequences.

MinION, a third-generation sequencer from Oxford Nanopore Technologies, is a portable device that can provide long-nucleotide read data in real-time. It primarily aims to deduce the makeup of nucleotide sequences from the ionic current signals generated when passing DNA/RNA fragments through nanopores charged with a voltage difference. To determine nucleotides from measured signals, a translation process known as basecalling is required. However, compared to NGS basecallers, the calling accuracy of MinION still needs to be improved. In this work, a simple but powerful neural network architecture called multi-scale recurrent caller (MSRCall) is proposed. MSRCall comprises a multi-scale structure, recurrent layers, a fusion block and a connectionist temporal classification decoder. To better identify both short-and long-range dependencies, the recurrent layer is redesigned to capture various time-scale features with a multi-scale structure. The results show that MSRCall outperforms other basecallers in terms of both read and consensus accuracies. MSRCall is available at: https://github.com/d05943006/MSRCall. Supplementary data are available at Bioinformatics online.

Attention-based CNN-ConvLSTM for Handwritten Arabic Word Extraction

Word extraction is one of the most critical steps in handwritten recognition systems. It is challenging for many reasons, such as the variability of handwritten writing styles, touching and overlapping characters, skewness problems, diacritics, ascenders, and descenders' presence. In this work, we propose a deep-learning-based approach for handwritten Arabic word extraction. We used an Attention-based CNN-ConvLSTM (Convolutional Long Short-term Memory) followed by a CTC (Connectionist Temporal Classification) function. Firstly, the text-line input image's essential features are extracted using Attention-based Convolutional Neural Networks (CNN). The extracted features and the text line's transcription are then passed to a ConvLSTM to learn a mapping between them. Finally, we used a CTC to learn the alignment between text-line images and their transcription automatically. We tested the proposed model on a complex dataset known as KFUPM Handwritten Arabic Text (KHATT \cite{khatt}). It consists of complex patterns of handwritten Arabic text-lines. The experimental results show an apparent efficiency of the used combination, where we ended up with an extraction success rate of 91.7\%.

Multiple attention-based encoder–decoder networks for gas meter character recognition

Factories swiftly and precisely grasp the real-time data of the production instrumentation, which is the foundation for the development and progress of industrial intelligence in industrial production. Weather, light, angle, and other unknown circumstances, on the other hand, impair the image quality of meter dials in natural environments, resulting in poor dial image quality. The remote meter reading system has trouble recognizing dial pictures in extreme settings, challenging it to meet industrial production demands. This paper provides multiple attention and encoder–decoder-based gas meter recognition networks (MAEDR) for this problem. First, from the acquired dial photos, the dial images with extreme conditions such as overexposure, artifacts, blurring, incomplete display of characters, and occlusion are chosen to generate the gas meter dataset. Then, a new character recognition network is proposed utilizing multiple attention and an encoder–decoder structure. Convolutional neural networks (CNN) extract visual features from dial images, encode visual features employing multi-head self-attention and position information, and facilitate feature alignment using the connectionist temporal classification (CTC) method. A novel two-step attention decoder is presented to improve the accuracy of recognition results. convolutional block attention module (CBAM) reweights the visual features from the CNN and the semantic features computed by the encoder to improve model performance; long short-term memory attention (LSTM attention) focuses on the relationship between feature sequences. According to experimental data, our system can effectively and efficiently identify industrial gas meter picture digits with 91.1% identification accuracy, faster inference speed, and higher accuracy than standard algorithms. The accuracy and practicality of the recognition can fulfill the needs of instrument data detection and recognition in industrial production, and it has a wide range of applications.

Decoupling music notation to improve end-to-end Optical Music Recognition

Inspired by the Text Recognition field, end-to-end schemes based on Convolutional Recurrent Neural Networks (CRNN) trained with the Connectionist Temporal Classification (CTC) loss function are considered one of the current state-of-the-art techniques for staff-level Optical Music Recognition (OMR). Unlike text symbols, music-notation elements may be defined as a combination of (i) a shape primitive located in (ii) a certain position in a staff. However, this double nature is generally neglected in the learning process, as each combination is treated as a single token. In this work, we study whether exploiting such particularity of music notation actually benefits the recognition performance and, if so, which approach is the most appropriate. For that, we thoroughly review existing specific approaches that explore this premise and propose different combinations of them. Furthermore, considering the limitations observed in such approaches, a novel decoding strategy specifically designed for OMR is proposed. The results obtained with four different corpora of historical manuscripts show the relevance of leveraging this double nature of music notation since it outperforms the standard approaches where it is ignored. In addition, the proposed decoding leads to significant reductions in the error rates with respect to the other cases.

Sequential Gesture Learning for Continuous Labanotation Generation Based on the Fusion of Graph Neural Networks

Labanotation is a symbolic recording system for human movements, and also a powerful tool for protecting and spreading folk dances and other performing arts. State-of-the-art automatic Labanotation uses end-to-end methods with sequence-based skeleton representation, which cannot capture the relationship between joints and bones in the skeleton for accurate descriptions of continuous lower limb movements such as dance steps. In this paper, we propose a novel double-stream fusion method of directed graph neural networks (DGNN), combined with connectionist temporal classification (CTC), namely DFGNN-CTC, for sequential fine-grained motion recognition, such as the Labanotation generation of unsegmented dance movement. First, we extract double-stream directed graph feature, employing an orientation-normalized directed acyclic graph (ON-DAG) and an orientation-normalized temporal directed acyclic graph (ON-TDAG), to jointly model spatiotemporal properties of movement recorded in motion capture data. Then, we design a CTC-based fusion-pooling module to fuse the spatial and temporal streams encoded by two DGNNs. It concatenates and fuses the two streams to generate discriminative descriptions of each time step, and concentrates them to make per-time-step predictions of Laban gesture type, from which the CTC searches the optimal Laban symbol sequence, corresponding to elemental motions composing the movement. In this way, the new method enables much finer discrimination for similar Laban gestures with subtle differences in spatial and temporal properties through joint contextual spatiotemporal modeling so that it achieves much superior performance in continuous Labanotation generation to existing methods, which only have single-stream analysis either spatially or temporally. The experiments on two Labanotation-labelled motion capture datasets demonstrate the effectiveness of the components in the proposed method and its superiority comparing with the state-of-the-art methods, especially for lower limb movements.

Self-supervised learning based knowledge distillation framework for automatic speech recognition for hearing impaired

The use of speech processing applications, particularly speech recognition, has got a lot of attention in recent decades. In recent years, research has focused on using deep learning for speech-related applications. This new branch of machine learning has outperformed others in a range of applications, including voice, and has thus become a particularly appealing research subject. Noise, speaker variability, language variability, vocabulary size, and domain remain one of the most significant research difficulties in speech recognition. We investigated on self-supervised algorithm for the unlabelled data. In recent years, these algorithms have progressed significantly, with their efficacy approaching and supervised pre-training alternatives across a variety of data modalities such as image and video. The purpose of this research is to develop powerful models for audio speech recognition that do not require human annotation. We accomplish this by distilling information from an automatic speech recognition (ASR) model that was trained on a large audio-only corpus. We integrate Connectionist Temporal Classification (CTC) loss, KL divergence loss in distillation technique. We demonstrate that distillation significantly speeds up training. We evaluate our model with evaluation metric Word Error Rate (WER).

A study of transformer-based end-to-end speech recognition system for Kazakh language

Today, the Transformer model, which allows parallelization and also has its own internal attention, has been widely used in the field of speech recognition. The great advantage of this architecture is the fast learning speed, and the lack of sequential operation, as with recurrent neural networks. In this work, Transformer models and an end-to-end model based on connectionist temporal classification were considered to build a system for automatic recognition of Kazakh speech. It is known that Kazakh is part of a number of agglutinative languages and has limited data for implementing speech recognition systems. Some studies have shown that the Transformer model improves system performance for low-resource languages. Based on our experiments, it was revealed that the joint use of Transformer and connectionist temporal classification models contributed to improving the performance of the Kazakh speech recognition system and with an integrated language model it showed the best character error rate 3.7% on a clean dataset.

DMS-SK/BLSTM-CTC Hybrid Network for Gesture/Speech Fusion and Its Application in Lunar Robot–Astronauts Interaction

In the future manned lunar exploration mission, astronauts would work with the lunar robots, which has a high requirement for human–robot interaction (HRI). As the accuracy of gesture recognition interaction does not fulfill the requirement for human–robot joint exploration missions, we propose the DMS-SK/BLSTM-CTC hybrid network to improve the performance of HRI. For gesture recognition, considering VGG-SK has low accuracy and complex architecture, we delete the fourth convolution module, optimize the last global pooling layer, introduce dilated convolution block and multiscale convolution block in VGG-SK, and get the DMS-SK-based gesture recognition sub-network. Compared with the traditional recognition methods, the accuracy and performance of DMS-SK improve. For speech recognition, considering that Bidirectional long–short-term memory unit (BLSTM) has the advantages of processing temporal information, and the Connectionist Temporal Classification (CTC) algorithm can simplify speech data preprocessing, we use BLSTM based on CTC as the speech recognition sub-network. Finally, we combine DMS-SK with BLSTM-CTC, and propose the DMS-SK/BLSTM-CTC hybrid network as the gesture/speech hybrid network. In addition, we use 10 gestures in the American Sign Language (ASL) dataset and 10 speech commands to construct the gesture/speech hybrid dataset. Experimental results show that compared with the pure gesture or pure speech networks, the recognition accuracy of the gesture-speech hybrid network improves by 2% and 12%, respectively, its accuracy reaches 97.38%, which fulfills the requirement of astronauts for HRI.

Domain Generalization for Language-Independent Automatic Speech Recognition.

A language-independent automatic speech recognizer (ASR) is one that can be used for phonetic transcription in languages other than the languages in which it was trained. Language-independent ASR is difficult to train, because different languages implement phones differently: even when phonemes in two different languages are written using the same symbols in the international phonetic alphabet, they are differentiated by different distributions of language-dependent redundant articulatory features. This article demonstrates that the goal of language-independence may be approximated in different ways, depending on the size of the training set, the presence vs. absence of familial relationships between the training and test languages, and the method used to implement phone recognition or classification. When the training set contains many languages, and when every language in the test set is related (shares the same language family with) a language in the training set, then language-independent ASR may be trained using an empirical risk minimization strategy (e.g., using connectionist temporal classification without extra regularizers). When the training set is limited to a small number of languages from one language family, however, and the test languages are not from the same language family, then the best performance is achieved by using domain-invariant representation learning strategies. Two different representation learning strategies are tested in this article: invariant risk minimization, and regret minimization. We find that invariant risk minimization is better at the task of phone token classification (given known segment boundary times), while regret minimization is better at the task of phone token recognition.

Automatic Speech Recognition Method Based on Deep Learning Approaches for Uzbek Language.

Communication has been an important aspect of human life, civilization, and globalization for thousands of years. Biometric analysis, education, security, healthcare, and smart cities are only a few examples of speech recognition applications. Most studies have mainly concentrated on English, Spanish, Japanese, or Chinese, disregarding other low-resource languages, such as Uzbek, leaving their analysis open. In this paper, we propose an End-To-End Deep Neural Network-Hidden Markov Model speech recognition model and a hybrid Connectionist Temporal Classification (CTC)-attention network for the Uzbek language and its dialects. The proposed approach reduces training time and improves speech recognition accuracy by effectively using CTC objective function in attention model training. We evaluated the linguistic and lay-native speaker performances on the Uzbek language dataset, which was collected as a part of this study. Experimental results show that the proposed model achieved a word error rate of 14.3% using 207 h of recordings as an Uzbek language training dataset.

End-to-End Sentence-Level Multi-View Lipreading Architecture with Spatial Attention Module Integrated Multiple CNNs and Cascaded Local Self-Attention-CTC

Concomitant with the recent advances in deep learning, automatic speech recognition and visual speech recognition (VSR) have received considerable attention. However, although VSR systems must identify speech from both frontal and profile faces in real-world scenarios, most VSR studies have focused solely on frontal face pictures. To address this issue, we propose an end-to-end sentence-level multi-view VSR architecture for faces captured from four different perspectives (frontal, 30°, 45°, and 60°). The encoder uses multiple convolutional neural networks with a spatial attention module to detect minor changes in the mouth patterns of similarly pronounced words, and the decoder uses cascaded local self-attention connectionist temporal classification to collect the details of local contextual information in the immediate vicinity, which results in a substantial performance boost and speedy convergence. To compare the performance of the proposed model for experiments on the OuluVS2 dataset, the dataset was divided into four different perspectives, and the obtained performance improvement was 3.31% (0°), 4.79% (30°), 5.51% (45°), 6.18% (60°), and 4.95% (mean), respectively, compared with the existing state-of-the-art performance, and the average performance improved by 9.1% compared with the baseline. Thus, the suggested design enhances the performance of multi-view VSR and boosts its usefulness in real-world applications.

Hand Witten Text Recognition using Deep Learning

Abstract: The dependence on people over technology has never been higher, to the point where deep learning and machine learning algorithms can conduct anything from object detection in images to adding sound to silent movies. Similarly, handwritten text recognition is a major field of research and development with a plethora of possibilities. Using this method, the system is taught to look for similarities as well as differences between diverse handwriting samples. This program turns a picture of a handwritten transcription into digital text. Convolution Neural Network (CNN) with the help of IAM databases are used on the proposed work to recognize the handwritten text. Our main goal is to integrate the aforementioned models, with their release times, in order to create the best text recognition possible, model. Keywords: Handwritten Text Recognition (HTR), Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Connectionist Temporal Classification (CTC)

Physiological Signal Emotion Recognition Based on Temporal Convolutional Networks

Abstract Human emotion recognition has become an important research field. Because of its objectivity, physiological signals have become one of the most robust cues for emotion recognition. In recent years, deep learning methods have made great progress in the field of emotion recognition, especially the superiority of recurrent neural networks (RNN) in time series models, and more and more tasks are completed based on RNN. However, RNN has problems such as time-consuming and gradient disappearance and explosion, and the feature input and sentiment output of RNN are not aligned. To avoid these problems, this paper is based on the temporal convolutional networks (TCN) model and the connectionist temporal classification (CTC) algorithm to process the emotion recognition task. First, generate a spectrogram representation of the physiological signal in each channel; Second, use the TCN to learn long-term dynamic features, and use CTC to align the dynamic features and their sentiment labels, and then feed the learned deep features into the neural network to predict the sentiment of each channel; Finally, take the best result as the final emotion representation. Experimental results on the AMIGOS dataset show that the proposed method outperforms existing methods.