Long-term Recurrent Convolutional Network Research Articles

Violence Detection in Jails and Mental Asylums

This research investigates the use of Long- term Recurrent Convolutional Networks (LRCNs) for violence detection in video surveillance systems. LRCNs combine the strengths of Convolutional Neural Networks (CNNs) for capturing spatial information and Long Short-Term Memory (LSTM) networks for modeling temporal sequences. This combination allows the system to learn complex spatiotemporal patterns in video data, improving violence detection accuracy in environments like jails and mental health facilities. The project focuses on the integration of the LRCN model with a Telegram bot for real-time alerting and response. Upon detecting violent incidents in the video streams, the LRCN model triggers alerts through the Telegram bot, providing instant notifications to relevant authorities. The Telegram bot facilitates seamless communication and coordination among stakeholders, enabling swift action to mitigate potential risks and ensure the safety of occupants within these facilities. Through rigorous experimentation and evaluation, the effectiveness and reliability of the LRCN-based violence detection system integrated with the Telegram bot are demonstrated. The research contributes to advancing technology-driven solutions for proactive security measures in high-risk environments, fostering safer and more secure institutional settings

Indonesian Lip-Reading Detection and Recognition Based on Lip Shape Using Face Mesh and Long-Term Recurrent Convolutional Network

Indonesian Lip-Reading Detection and Recognition Based on Lip Shape Using Face Mesh and Long-Term Recurrent Convolutional Network

Eye-LRCN: A Long-Term Recurrent Convolutional Network for Eye Blink Completeness Detection.

Computer vision syndrome causes vision problems and discomfort mainly due to dry eye. Several studies show that dry eye in computer users is caused by a reduction in the blink rate and an increase in the prevalence of incomplete blinks. In this context, this article introduces Eye-LRCN, a new eye blink detection method that also evaluates the completeness of the blink. The method is based on a long-term recurrent convolutional network (LRCN), which combines a convolutional neural network (CNN) for feature extraction with a bidirectional recurrent neural network that performs sequence learning and classifies the blinks. A Siamese architecture is used during CNN training to overcome the high-class imbalance present in blink detection and the limited amount of data available to train blink detection models. The method was evaluated on three different tasks: blink detection, blink completeness detection, and eye state detection. We report superior performance to the state-of-the-art methods in blink detection and blink completeness detection, and remarkable results in eye state detection.

Sign Language Word Detection Using LRCN

Sign language is the most effective communication for deaf or hard-of-hearing people. Specialized training is required to understand sign language, and as such, people without disabilities around them cannot communicate effectively. The main objective of this study is to develop a mechanism for streamlining the deep learning model for sign language recognition by utilizing the 30 most prevalent words in our everyday lives. The dataset was designed through 30 ASL (American Sign Language) words consisting of custom-processed video sequences, which consist of 5 subjects and 50 sample videos for each class. The CNN model can be applied to video frames to extract spatial properties. Using CNN’s acquired data, the LSTM model may then predict the action being performed in the video. We present and evaluate the results of two separate datasets—the Pose dataset and the Raw video dataset. The dataset was trained with the Long-term Recurrent Convolutional Network (LRCN) approach. Finally, a test accuracy of 92.66% was reached for the raw dataset, while 93.66% for the pose dataset.

Investigating the Influence of Convolutional Operations on LSTM Networks in Video Classification

Video classification holds a foundational position in the realm of computer vision, involving the categorization and labeling of videos based on their content. Its significance resonates across various applications, including video surveil-lance, content recommendation, action recognition, video indexing, and more. The primary objective of video classification is to automatically analyze and comprehend the visual information embedded in videos, facilitating the efficient organization, retrieval, and interpretation of extensive video collections. The integration of convolutional neural networks (CNNs) and long short-term memory (LSTM) networks has brought about a revolution in video classification. This fusion effectively captures both spatial and temporal dependencies within video sequences, leveraging the strengths of CNNs in extracting spatial features and LSTMs in modeling sequential and temporal information. ConvLSTM and LRCN (Long-term Recurrent Convolutional Networks) are two widely embraced architectures that embody this fusion. This paper seeks to investigate the impact of convolutions on LSTM networks in the context of video classification, aiming to compare the performance of ConvLSTM and LRCN.

TAEKWONDO POSE ESTIMATION WITH DEEP LEARNING ARCHITECTURES ON ONE-DIMENSIONAL AND TWO-DIMENSIONAL DATA

Practicing sports is an activity that helps people maintain and improve their health, enhance memory and concentration, reduce anxiety and stress, and train teamwork and leadership ability. With the development of science and technology, artificial intelligence in sports has become increasingly popular with the public and brings many benefits. In particular, many applications help people track and evaluate athletes' achievements in competitions. This study extracts images from Taekwondo videos and generates skeleton data from frames using the Fast Forward Moving Picture Experts Group (FFMPEG) technique using MoveNet. After that, we use deep learning architectures such as Long Short-Term Memory Networks, Convolutional Long Short-Term Memory, and Long-term Recurrent Convolutional Networks to perform the poses classification tasks in Taegeuk in Jang lessons. This work presents two approaches. The first approach uses a sequence skeleton extracted from the image by Movenet. Second, we use sequence images to train using video classification architecture. Finally, we recognize poses in sports lessons using skeleton data to remove noise in the image, such as background and extraneous objects behind the exerciser. As a result, our proposed method has achieved promising performance in pose classification tasks in an introductory Taekwondo lesson.

Analysis and comparison of various deep learning models to implement suspicious activity recognition in CCTV surveillance

The paper aims to analyze and compare various deep learning (DL) algorithms in order to develop a Suspicious Activity Recognition (SAR) system for closed-circuit television (CCTV) surveillance. Automated systems for detecting and classifying suspicious activities are crucial as technology’s role in safety and security expands. This paper addresses these challenges by creating a robust SAR system using machine learning techniques. It analyzes and compares evaluation metrics such as Precision, Recall, F1 Score, and Accuracy using various deep learning methods (convolutional neural network (CNN), Long short-term memory (LSTM) – Visual Geometry Group 16 (VGG16), LSTM – ResNet50, LSTM – EfficientNetB0, LSTM – InceptionNetV3, LSTM – DenseNet121, and Long-term Recurrent Convolutional Network (LRCN)). The proposed system improves threat identification, vandalism deterrence, fight prevention, and video surveillance. It aids emergency response by accurately classifying suspicious activities from CCTV footage, reducing reliance on human security personnel and addressing limitations in manual monitoring. The objectives of the paper include analyzing existing works, extracting features from CCTV videos, training robust deep learning models, evaluating algorithms, and improving accuracy. The conclusion highlights the superior performance of the LSTM-DenseNet121 algorithm, achieving an overall accuracy of 91.17% in detecting suspicious activities. This enhances security monitoring capabilities and reduces response time. Limitations of the system include subjectivity, contextual understanding, occlusion, false alarms, and privacy concerns. Future improvements involve real-time object tracking, collaboration with law enforcement agencies, and performance optimization. Ongoing research is necessary to overcome limitations and enhance the effectiveness of CCTV surveillance.

Deep Learning Algorithms for Behavioral Analysis in Diagnosing Neurodevelopmental Disorders

Autism spectrum disorder (ASD), or autism, can be diagnosed based on a lack of behavioral skills and social communication. The most prominent method of diagnosing ASD in children is observing the child’s behavior, including some of the signs that the child repeats. Hand flapping is a common stimming behavior in children with ASD. This research paper aims to identify children’s abnormal behavior, which might be a sign of autism, using videos recorded in a natural setting during the children’s regular activities. Specifically, this study seeks to classify self-stimulatory activities, such as hand flapping, as well as normal behavior in real-time. Two deep learning video classification methods are used to be trained on the publicly available Self-Stimulatory Behavior Dataset (SSBD). The first method is VGG-16-LSTM; VGG-16 to spatial feature extraction and long short-term memory networks (LSTM) for temporal features. The second method is a long-term recurrent convolutional network (LRCN) that learns spatial and temporal features immediately in end-to-end training. The VGG-16-LSTM achieved 0.93% on the testing set, while the LRCN model achieved an accuracy of 0.96% on the testing set.

Autonomous assessment of spontaneous retinal venous pulsations in fundus videos using a deep learning framework

The presence or absence of spontaneous retinal venous pulsations (SVP) provides clinically significant insight into the hemodynamic status of the optic nerve head. Reduced SVP amplitudes have been linked to increased intracranial pressure and glaucoma progression. Currently, monitoring for the presence or absence of SVPs is performed subjectively and is highly dependent on trained clinicians. In this study, we developed a novel end-to-end deep model, called U3D-Net, to objectively classify SVPs as present or absent based on retinal fundus videos. The U3D-Net architecture consists of two distinct modules: an optic disc localizer and a classifier. First, a fast attention recurrent residual U-Net model is applied as the optic disc localizer. Then, the localized optic discs are passed on to a deep convolutional network for SVP classification. We trained and tested various time-series classifiers including 3D Inception, 3D Dense-ResNet, 3D ResNet, Long-term Recurrent Convolutional Network, and ConvLSTM. The optic disc localizer achieved a dice score of 95% for locating the optic disc in 30 milliseconds. Amongst the different tested models, the 3D Inception model achieved an accuracy, sensitivity, and F1-Score of 84 ± 5%, 90 ± 8%, and 81 ± 6% respectively, outperforming the other tested models in classifying SVPs. To the best of our knowledge, this research is the first study that utilizes a deep neural network for an autonomous and objective classification of SVPs using retinal fundus videos.

A Long-Term Recurrent Convolutional Network for SNR Estimation of Cone-Shaped Target

Signal-to-Noise Ratio (SNR) is an important prior information for many micro-motion echo processing approaches and techniques. Estimating SNR in advance can effectively enhance the performance of such technologies. This letter proposes a Long Term Recurrent Convolutional Network (LRCN) based SNR estimation method for cone-shaped target. First, mathematical expression of the echo is derived by studying the cone-shaped target micro-motion model, and hence the characteristics of the echo and its Short-Time Fourier Transform (STFT) are analyzed. Second, the motivation for the proposed are analyzed, and the echo is converted as an time-frequency graph (TFG) with STFT, so that an estimation technique based on LRCN (a hybrid of convolutional neural network (CNN) and recurrent neural networks (RNN)) is designed to estimate the SNR. Experiments indicate that the proposed method outperforms other already existing methods with better accuracy and robustness.

Multi-classification of kidney abnormalities in sonography using the LOA-MFO and long-term recurrent convolutional network

Multi-classification of kidney abnormalities in sonography using the LOA-MFO and long-term recurrent convolutional network

Weighted knowledge distillation of attention-LRCN for recognizing affective states from PPG signals

The recognition of affective states is important for regulating stress levels and maintaining mental health, and it is known that affective states can be inferred from physiological signals. However, in practice, the problem of affect recognition contains many challenges due to various types of external noise and different individual characteristics. This paper proposes a deep learning model called Attention-LRCN for recognizing affective states from photoplethysmography (PPG) signals. We construct a long-term recurrent convolutional network to extract temporal features from spectrograms, and a novel attention module is introduced to alleviate the effect of noise components in PPG signals. Moreover, to improve the recognition accuracy, we propose a weighted knowledge distillation technique, which is a teacher–student learning framework. We quantify the uncertainty of teacher’s predictions, and the predictive uncertainty is utilized to adaptively compute the weight of the distillation loss. To demonstrate the effectiveness of the proposed method, experiments were conducted on the WESAD dataset, which is a public dataset for stress and affect detection. We also collected our own dataset from 34 subjects to verify the accuracy of the proposed method. Experimental results demonstrate that the proposed method significantly outperforms previous algorithms on both the public and real-world datasets. The code is available at https://github.com/ziiho08/Attention-LRCN.

Academic Emotion Classification Using FER: A Systematic Review

Facial emotion expressions are among the most potent, natural, and powerful means of human communication. Due to the COVID-19 pandemic, educational institutions worldwide are forced to switch rapidly to remote and online learning. Students are currently in an emergency state and must adapt to various and readily accessible learning methods, such as mobile learning applications or an e-learning system. A systematic literature review (SLR) is conducted to extract and synthesize information such as the emotion classifier used in the facial expression recognition (FER) system, the dataset used, the preprocessing technique applied, the feature extraction approach used, and the strength and limitation of the previous studies. Based on the search criteria, 701 publications were initially retrieved from five different digital databases, of which 48 studies have been chosen as primary studies for further analysis. Based on the findings of this study, the deep learning approach is the most frequently adopted approach in classifying student emotions during online learning. FER-2013 is the most commonly used FER dataset in FER studies, while DAiSEE is the most used academic emotion dataset. Moreover, support vector machine (SVM) is the conventional learning emotion classifier that is widely used in the FER systems, while convolutional neural network (CNN) is the most frequently used deep learning classifier. Next, it was found that the number of real-time FER systems is less than that of non-real-time FER systems. Finally, the top-1 accuracy of 94.6% was achieved by the long-term recurrent convolutional network on the academic emotion dataset, and the limitation is that it has low illumination and a lack of frontal pose.

Recognizing Human Activity Using Hybrid Models of CNN and LSTM in Deep Learning

Human Activity Recognition (HAR) is a time series categorization challenge that requires data from a number of timesteps in order to correctly classify the activities that are carried out. In recent times, the usage of image datasets for activity recognition has increased, however good classification cannot be done with just one frame. To increase recognition accuracy, multiple frames of data and the context of environment are required. It is known that a video is made up of a number of still images (frames) that are quickly updated to create the illusion of motion. The hybrid models of Deep Learning (DL) algorithms like Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) are proposed for recognising the human activity from video dataset. The hybrid models, Convolutional Long Short-Term Memory (ConvLSTM) and Long-term Recurrent Convolutional Network (LRCN) are introduced to improve the accuracy of HAR on video dataset. The models will be evaluated on standard video datasets, and the outcomes will show how HAR has the potential to significantly influence a number of industries. It has many applications in the fields of security, sports, and healthcare.

Operational solar flare forecasting via video-based deep learning

Operational flare forecasting aims at providing predictions that can be used to make decisions, typically on a daily scale, about the space weather impacts of flare occurrence. This study shows that video-based deep learning can be used for operational purposes when the training and validation sets used for network optimization are generated while accounting for the periodicity of the solar cycle. Specifically, this article describes an algorithm that can be applied to build up sets of active regions that are balanced according to the flare class rates associated to a specific cycle phase. These sets are used to train and validate a long-term recurrent convolutional network made of a combination of a convolutional neural network and a long short-term memory network. The reliability of this approach is assessed in the case of two prediction windows containing the solar storms of March 2015, June 2015, and September 2017.

Deep-Learning-Based Sequence Causal Long-Term Recurrent Convolutional Network for Data Fusion Using Video Data

The purpose of AI-Based schemes in intelligent systems is to advance and optimize system performance. Most intelligent systems adopt sequential data types derived from such systems. Realtime video data, for example, are continuously updated as a sequence to make necessary predictions for efficient system performance. The majority of deep-learning-based network architectures such as long short-term memory (LSTM), data fusion, two streams, and temporal convolutional network (TCN) for sequence data fusion are generally used to enhance robust system efficiency. In this paper, we propose a deep-learning-based neural network architecture for non-fix data that uses both a causal convolutional neural network (CNN) and a long-term recurrent convolutional network (LRCN). Causal CNNs and LRCNs use incorporated convolutional layers for feature extraction, so both architectures are capable of processing sequential data such as time series or video data that can be used in a variety of applications. Both architectures also have extracted features from the input sequence data to reduce the dimensionality of the data and capture the important information, and learn hierarchical representations for effective sequence processing tasks. We have also adopted a concept of series compact convolutional recurrent neural network (SCCRNN), which is a type of neural network architecture designed for processing sequential data combined by both convolutional and recurrent layers compactly, reducing the number of parameters and memory usage to maintain high accuracy. The architecture is challenge-able and suitable for continuously incoming sequence video data, and doing so allowed us to bring advantages to both LSTM-based networks and CNNbased networks. To verify this method, we evaluated it through a sequence learning model with network parameters and memory that are required in real environments based on the UCF-101 dataset, which is an action recognition data set of realistic action videos, collected from YouTube with 101 action categories. The results show that the proposed model in a sequence causal long-term recurrent convolutional network (SCLRCN) provides a performance improvement of at least 12% approximately or more to be compared with the existing models (LRCN and TCN).

Online Learning Engagement Recognition Using Bidirectional Long-Term Recurrent Convolutional Networks

Background: Online learning is currently adopted by educational institutions worldwide to provide students with ongoing education during the COVID-19 pandemic. However, online learning has seen students lose interest and become anxious, which affects learning performance and leads to dropout. Thus, measuring students’ engagement in online learning has become imperative. It is challenging to recognize online learning engagement due to the lack of effective recognition methods and publicly accessible datasets. Methods: This study gathered a large number of online learning videos of students at a normal university. Engagement cues were used to annotate the dataset, which was constructed with three levels of engagement: low engagement, engagement, and high engagement. Then, we introduced a bi-directional long-term recurrent convolutional network (BiLRCN) for online learning engagement recognition in video. Result: An online learning engagement dataset has been constructed. We evaluated six methods using precision and recall, where BiLRCN obtained the best performance. Conclusions: Both category balance and category similarity of the data affect the performance of the results; it is more appropriate to consider learning engagement as a process-based evaluation; learning engagement can provide intervention strategies for teachers from a variety of perspectives and is associated with learning performance. Dataset construction and deep learning methods need to be improved, and learning data management also deserves attention.

Wholesale Electricity Price Forecasting Using Integrated Long-Term Recurrent Convolutional Network Model

Electricity price forecasts have become a fundamental factor affecting the decision-making of all market participants. Extreme price volatility has forced market participants to hedge against volume risks and price movements. Hence, getting an accurate price forecast from a few hours to a few days ahead is very important and very challenging due to various factors. This paper proposes an integrated long-term recurrent convolutional network (ILRCN) model to predict electricity prices considering the majority of contributing attributes to the market price as input. The proposed ILRCN model combines the functionalities of a convolutional neural network and long short-term memory (LSTM) algorithm along with the proposed novel conditional error correction term. The combined ILRCN model can identify the linear and nonlinear behavior within the input data. ERCOT wholesale market price data along with load profile, temperature, and other factors for the Houston region have been used to illustrate the proposed model. The performance of the proposed ILRCN electricity price forecasting model is verified using performance/evaluation metrics like mean absolute error and accuracy. Case studies reveal that the proposed ILRCN model shows the highest accuracy and efficiency in electricity price forecasting as compared to the support vector machine (SVM) model, fully connected neural network model, LSTM model, and the traditional LRCN model without the conditional error correction stage.

Video Based Human Activity Recognition Surveillance System

It is argued that current state-of-the-art methods of home surveillance such as motion detection technology like CCTV intrusion alert are insufficient, in particular to cater the modern need of whole automation with vulnerabilities such as requiring human involvement. We propose an alternative system, a video-based Human Activity Recognition (HAR) approach using the combination of Convolution Neural Network (CNN) and Long Short-Term Memory (LSTM) algorithm, to dealwith the identified shortcomings. Our proposal does not require any changes to the existing home security protocols and is easily implemented using only low-cost, commercial-off-the-shelf hardware. We can easily use the traditional surveillance camera for computer vision tasks. We evaluate our approach using real-world activity data collected via video-based sensor. We quantify its effectiveness, by plotting Loss and accuracy curves. Our results show that the video-based HAR approach can provide full automation in home surveillancesystem compared to conventional CCTV motion detectors by an accuracy of more than 93%. Further system’s accuracy can be increased and we can achieve significantly better results by implementing LRCN (Long Term Recurrent Convolution Network) approach.

Singing Voice Detection in Electronic Music with a Long-Term Recurrent Convolutional Network

Singing Voice Detection (SVD) is a classification task that determines whether there is a singing voice in a given audio segment. While current systems produce high-quality results on this task, the reported experiments are usually limited to popular music. A Long-Term Recurrent Convolutional Network (LRCN) was adapted to detect vocals in a new dataset of electronic music to evaluate its performance in a different music genre and compare its results against those in other state-of-the-art experiments in pop music to prove its effectiveness across a different genre. Experiments on two datasets studied the impacts of different audio features and block size on LRCN temporal relationship learning, and the benefits of preprocessing on performance, and the results generate a benchmark to evaluate electronic music and its intricacies.