Long-term Recurrent Convolutional Network Research Articles

Deep learning-based video surveillance system for suspicious activity detection

The need for reliable video surveillance systems to detect and prevent suspicious activities has become more important with the increase in crime and security threats. This paper proposes a real-time video surveillance system based on the Long-term Recurrent Convolutional Network (LRCN) model, which can automatically detect and alert the authority about suspicious activities, such as fighting, accidents, and robbery. Our system comprises two main components: LRCN-based activity recognition and real-time alert generation. We evaluated the performance of the proposed system on a custom dataset compiled from two publicly available datasets and achieved state-of-the-art results in terms of accuracy, precision, and recall. Our results demonstrate the effectiveness and scalability of the LRCN-based video surveillance system for real-time suspicious activity detection. We believe that our proposed system can be deployed in various public places, such as airports, train stations, and shopping malls, to enhance the security and safety of the public.

Multi-Directional Long-Term Recurrent Convolutional Network for Road Situation Recognition.

Understanding road conditions is essential for implementing effective road safety measures and driving solutions. Road situations encompass the day-to-day conditions of roads, including the presence of vehicles and pedestrians. Surveillance cameras strategically placed along streets have been instrumental in monitoring road situations and providing valuable information on pedestrians, moving vehicles, and objects within road environments. However, these video data and information are stored in large volumes, making analysis tedious and time-consuming. Deep learning models are increasingly utilized to monitor vehicles and identify and evaluate road and driving comfort situations. However, the current neural network model requires the recognition of situations using time-series video data. In this paper, we introduced a multi-directional detection model for road situations to uphold high accuracy. Deep learning methods often integrate long short-term memory (LSTM) into long-term recurrent network architectures. This approach effectively combines recurrent neural networks to capture temporal dependencies and convolutional neural networks (CNNs) to extract features from extensive video data. In our proposed method, we form a multi-directional long-term recurrent convolutional network approach with two groups equipped with CNN and two layers of LSTM. Additionally, we compare road situation recognition using convolutional neural networks, long short-term networks, and long-term recurrent convolutional networks. The paper presents a method for detecting and recognizing multi-directional road contexts using a modified LRCN. After balancing the dataset through data augmentation, the number of video files increased, resulting in our model achieving 91% accuracy, a significant improvement from the original dataset.

Human Action Recognition using Long Short-Term Memory and Convolutional Neural Network Model

Human Action Recognition (HAR) is the difficulty of quickly identifying strenuous exercise performed by people. It is feasible to sample some measures of a body's tangential acceleration and speed using inertial sensors and exercise them only to learn model skills of incorrectly categorizing behavior into the relevant categories. In detecting human activities, the use of detectors in personal and portable devices has increased to better understand and anticipate human behavior. Many specialists are working toward developing a classification that can distinguish between a user's behavior and uncooked data while utilizing as few reserves as possible. A Long-term Recurrent Convolutional Network (LRCN) is proposed as a comprehensive human action recognition system based on deep neural networks in this paper.

Violence Detection in Jails and Mental Asylums

This research investigates the use of Long- term Recurrent Convolutional Networks (LRCNs) for violence detection in video surveillance systems. LRCNs combine the strengths of Convolutional Neural Networks (CNNs) for capturing spatial information and Long Short-Term Memory (LSTM) networks for modeling temporal sequences. This combination allows the system to learn complex spatiotemporal patterns in video data, improving violence detection accuracy in environments like jails and mental health facilities. The project focuses on the integration of the LRCN model with a Telegram bot for real-time alerting and response. Upon detecting violent incidents in the video streams, the LRCN model triggers alerts through the Telegram bot, providing instant notifications to relevant authorities. The Telegram bot facilitates seamless communication and coordination among stakeholders, enabling swift action to mitigate potential risks and ensure the safety of occupants within these facilities. Through rigorous experimentation and evaluation, the effectiveness and reliability of the LRCN-based violence detection system integrated with the Telegram bot are demonstrated. The research contributes to advancing technology-driven solutions for proactive security measures in high-risk environments, fostering safer and more secure institutional settings

Eye-LRCN: A Long-Term Recurrent Convolutional Network for Eye Blink Completeness Detection.

Computer vision syndrome causes vision problems and discomfort mainly due to dry eye. Several studies show that dry eye in computer users is caused by a reduction in the blink rate and an increase in the prevalence of incomplete blinks. In this context, this article introduces Eye-LRCN, a new eye blink detection method that also evaluates the completeness of the blink. The method is based on a long-term recurrent convolutional network (LRCN), which combines a convolutional neural network (CNN) for feature extraction with a bidirectional recurrent neural network that performs sequence learning and classifies the blinks. A Siamese architecture is used during CNN training to overcome the high-class imbalance present in blink detection and the limited amount of data available to train blink detection models. The method was evaluated on three different tasks: blink detection, blink completeness detection, and eye state detection. We report superior performance to the state-of-the-art methods in blink detection and blink completeness detection, and remarkable results in eye state detection.

Sign Language Word Detection Using LRCN

Sign language is the most effective communication for deaf or hard-of-hearing people. Specialized training is required to understand sign language, and as such, people without disabilities around them cannot communicate effectively. The main objective of this study is to develop a mechanism for streamlining the deep learning model for sign language recognition by utilizing the 30 most prevalent words in our everyday lives. The dataset was designed through 30 ASL (American Sign Language) words consisting of custom-processed video sequences, which consist of 5 subjects and 50 sample videos for each class. The CNN model can be applied to video frames to extract spatial properties. Using CNN’s acquired data, the LSTM model may then predict the action being performed in the video. We present and evaluate the results of two separate datasets—the Pose dataset and the Raw video dataset. The dataset was trained with the Long-term Recurrent Convolutional Network (LRCN) approach. Finally, a test accuracy of 92.66% was reached for the raw dataset, while 93.66% for the pose dataset.

Deep learning-based human activity recognition using CNN, ConvLSTM, and LRCN

Human activity recognition (HAR) plays a crucial role in assisting the elderly and individuals with vascular dementia by providing support and monitoring for their daily activities. This paper presents a deep learning (DL)-based approach to HAR, leveraging convolutional neural network (CNN), convolutional long short-term memory (ConvLSTM), and long-term recurrent convolutional network (LRCN) architectures. These models are designed to extract spatial features and capture temporal dependencies in video data, enhancing the accuracy of activity classification. We conducted experiments on the UCF50 and HMDB51 video datasets, encompassing diverse human activities. Our evaluation demonstrates that the ConvLSTM model achieves an accuracy of 82% on UCF50 and 68% on HMDB51, while the LRCN model gives accuracies of 93.44% and 71.55%, respectively. Finally, the CNN model outperforms with an accuracy rate of 99.58% for the UCF50 and 92.70% for the HMDB51 datasets. These significant improvements showcase the effectiveness of integrating convolutional and recurrent neural networks for HAR tasks. Our research contributes to advancing HAR systems with potential applications in healthcare, assisted living, and surveillance. By accurately recognizing human activities, our models can assist in remote patient monitoring, fall detection, and public safety initiatives. These findings underscore the importance of DL in enhancing the quality of life and safety for individuals in various contexts.

Intelligent diagnosis system based on artificial intelligence models for predicting freezing of gait in Parkinson's disease.

Freezing of gait (FoG) is a significant issue for those with Parkinson's disease (PD) since it is a primary contributor to falls and is linked to a poor superiority of life. The underlying apparatus is still not understood; however, it is postulated that it is associated with cognitive disorders, namely impairments in executive and visuospatial functions. During episodes of FoG, patients may experience the risk of falling, which significantly effects their quality of life. This research aims to systematically evaluate the effectiveness of machine learning approaches in accurately predicting a FoG event before it occurs. The system was tested using a dataset collected from the Kaggle repository and comprises 3D accelerometer data collected from the lower backs of people who suffer from episodes of FoG, a severe indication frequently realized in persons with Parkinson's disease. Data were acquired by measuring acceleration from 65 patients and 20 healthy senior adults while they engaged in simulated daily life tasks. Of the total participants, 45 exhibited indications of FoG. This research utilizes seven machine learning methods, namely the decision tree, random forest, Knearest neighbors algorithm, LightGBM, and CatBoost models. The Gated Recurrent Unit (GRU)-Transformers and Longterm Recurrent Convolutional Networks (LRCN) models were applied to predict FoG. The construction and model parameters were planned to enhance performance by mitigating computational difficulty and evaluation duration. The decision tree exhibited exceptional performance, achieving sensitivity rates of 91% in terms of accuracy, precision, recall, and F1- score metrics for the FoG, transition, and normal activity classes, respectively. It has been noted that the system has the capacity to anticipate FoG objectively and precisely. This system will be instrumental in advancing consideration in furthering the comprehension and handling of FoG.

Investigating the Influence of Convolutional Operations on LSTM Networks in Video Classification

Video classification holds a foundational position in the realm of computer vision, involving the categorization and labeling of videos based on their content. Its significance resonates across various applications, including video surveil-lance, content recommendation, action recognition, video indexing, and more. The primary objective of video classification is to automatically analyze and comprehend the visual information embedded in videos, facilitating the efficient organization, retrieval, and interpretation of extensive video collections. The integration of convolutional neural networks (CNNs) and long short-term memory (LSTM) networks has brought about a revolution in video classification. This fusion effectively captures both spatial and temporal dependencies within video sequences, leveraging the strengths of CNNs in extracting spatial features and LSTMs in modeling sequential and temporal information. ConvLSTM and LRCN (Long-term Recurrent Convolutional Networks) are two widely embraced architectures that embody this fusion. This paper seeks to investigate the impact of convolutions on LSTM networks in the context of video classification, aiming to compare the performance of ConvLSTM and LRCN.

TAEKWONDO POSE ESTIMATION WITH DEEP LEARNING ARCHITECTURES ON ONE-DIMENSIONAL AND TWO-DIMENSIONAL DATA

Practicing sports is an activity that helps people maintain and improve their health, enhance memory and concentration, reduce anxiety and stress, and train teamwork and leadership ability. With the development of science and technology, artificial intelligence in sports has become increasingly popular with the public and brings many benefits. In particular, many applications help people track and evaluate athletes' achievements in competitions. This study extracts images from Taekwondo videos and generates skeleton data from frames using the Fast Forward Moving Picture Experts Group (FFMPEG) technique using MoveNet. After that, we use deep learning architectures such as Long Short-Term Memory Networks, Convolutional Long Short-Term Memory, and Long-term Recurrent Convolutional Networks to perform the poses classification tasks in Taegeuk in Jang lessons. This work presents two approaches. The first approach uses a sequence skeleton extracted from the image by Movenet. Second, we use sequence images to train using video classification architecture. Finally, we recognize poses in sports lessons using skeleton data to remove noise in the image, such as background and extraneous objects behind the exerciser. As a result, our proposed method has achieved promising performance in pose classification tasks in an introductory Taekwondo lesson.

Analysis and comparison of various deep learning models to implement suspicious activity recognition in CCTV surveillance

The paper aims to analyze and compare various deep learning (DL) algorithms in order to develop a Suspicious Activity Recognition (SAR) system for closed-circuit television (CCTV) surveillance. Automated systems for detecting and classifying suspicious activities are crucial as technology’s role in safety and security expands. This paper addresses these challenges by creating a robust SAR system using machine learning techniques. It analyzes and compares evaluation metrics such as Precision, Recall, F1 Score, and Accuracy using various deep learning methods (convolutional neural network (CNN), Long short-term memory (LSTM) – Visual Geometry Group 16 (VGG16), LSTM – ResNet50, LSTM – EfficientNetB0, LSTM – InceptionNetV3, LSTM – DenseNet121, and Long-term Recurrent Convolutional Network (LRCN)). The proposed system improves threat identification, vandalism deterrence, fight prevention, and video surveillance. It aids emergency response by accurately classifying suspicious activities from CCTV footage, reducing reliance on human security personnel and addressing limitations in manual monitoring. The objectives of the paper include analyzing existing works, extracting features from CCTV videos, training robust deep learning models, evaluating algorithms, and improving accuracy. The conclusion highlights the superior performance of the LSTM-DenseNet121 algorithm, achieving an overall accuracy of 91.17% in detecting suspicious activities. This enhances security monitoring capabilities and reduces response time. Limitations of the system include subjectivity, contextual understanding, occlusion, false alarms, and privacy concerns. Future improvements involve real-time object tracking, collaboration with law enforcement agencies, and performance optimization. Ongoing research is necessary to overcome limitations and enhance the effectiveness of CCTV surveillance.

Deep Learning Algorithms for Behavioral Analysis in Diagnosing Neurodevelopmental Disorders

Autism spectrum disorder (ASD), or autism, can be diagnosed based on a lack of behavioral skills and social communication. The most prominent method of diagnosing ASD in children is observing the child’s behavior, including some of the signs that the child repeats. Hand flapping is a common stimming behavior in children with ASD. This research paper aims to identify children’s abnormal behavior, which might be a sign of autism, using videos recorded in a natural setting during the children’s regular activities. Specifically, this study seeks to classify self-stimulatory activities, such as hand flapping, as well as normal behavior in real-time. Two deep learning video classification methods are used to be trained on the publicly available Self-Stimulatory Behavior Dataset (SSBD). The first method is VGG-16-LSTM; VGG-16 to spatial feature extraction and long short-term memory networks (LSTM) for temporal features. The second method is a long-term recurrent convolutional network (LRCN) that learns spatial and temporal features immediately in end-to-end training. The VGG-16-LSTM achieved 0.93% on the testing set, while the LRCN model achieved an accuracy of 0.96% on the testing set.

Autonomous assessment of spontaneous retinal venous pulsations in fundus videos using a deep learning framework

The presence or absence of spontaneous retinal venous pulsations (SVP) provides clinically significant insight into the hemodynamic status of the optic nerve head. Reduced SVP amplitudes have been linked to increased intracranial pressure and glaucoma progression. Currently, monitoring for the presence or absence of SVPs is performed subjectively and is highly dependent on trained clinicians. In this study, we developed a novel end-to-end deep model, called U3D-Net, to objectively classify SVPs as present or absent based on retinal fundus videos. The U3D-Net architecture consists of two distinct modules: an optic disc localizer and a classifier. First, a fast attention recurrent residual U-Net model is applied as the optic disc localizer. Then, the localized optic discs are passed on to a deep convolutional network for SVP classification. We trained and tested various time-series classifiers including 3D Inception, 3D Dense-ResNet, 3D ResNet, Long-term Recurrent Convolutional Network, and ConvLSTM. The optic disc localizer achieved a dice score of 95% for locating the optic disc in 30 milliseconds. Amongst the different tested models, the 3D Inception model achieved an accuracy, sensitivity, and F1-Score of 84 ± 5%, 90 ± 8%, and 81 ± 6% respectively, outperforming the other tested models in classifying SVPs. To the best of our knowledge, this research is the first study that utilizes a deep neural network for an autonomous and objective classification of SVPs using retinal fundus videos.

A Long-Term Recurrent Convolutional Network for SNR Estimation of Cone-Shaped Target

Signal-to-Noise Ratio (SNR) is an important prior information for many micro-motion echo processing approaches and techniques. Estimating SNR in advance can effectively enhance the performance of such technologies. This letter proposes a Long Term Recurrent Convolutional Network (LRCN) based SNR estimation method for cone-shaped target. First, mathematical expression of the echo is derived by studying the cone-shaped target micro-motion model, and hence the characteristics of the echo and its Short-Time Fourier Transform (STFT) are analyzed. Second, the motivation for the proposed are analyzed, and the echo is converted as an time-frequency graph (TFG) with STFT, so that an estimation technique based on LRCN (a hybrid of convolutional neural network (CNN) and recurrent neural networks (RNN)) is designed to estimate the SNR. Experiments indicate that the proposed method outperforms other already existing methods with better accuracy and robustness.

Multi-classification of kidney abnormalities in sonography using the LOA-MFO and long-term recurrent convolutional network

Multi-classification of kidney abnormalities in sonography using the LOA-MFO and long-term recurrent convolutional network

Weighted knowledge distillation of attention-LRCN for recognizing affective states from PPG signals

The recognition of affective states is important for regulating stress levels and maintaining mental health, and it is known that affective states can be inferred from physiological signals. However, in practice, the problem of affect recognition contains many challenges due to various types of external noise and different individual characteristics. This paper proposes a deep learning model called Attention-LRCN for recognizing affective states from photoplethysmography (PPG) signals. We construct a long-term recurrent convolutional network to extract temporal features from spectrograms, and a novel attention module is introduced to alleviate the effect of noise components in PPG signals. Moreover, to improve the recognition accuracy, we propose a weighted knowledge distillation technique, which is a teacher–student learning framework. We quantify the uncertainty of teacher’s predictions, and the predictive uncertainty is utilized to adaptively compute the weight of the distillation loss. To demonstrate the effectiveness of the proposed method, experiments were conducted on the WESAD dataset, which is a public dataset for stress and affect detection. We also collected our own dataset from 34 subjects to verify the accuracy of the proposed method. Experimental results demonstrate that the proposed method significantly outperforms previous algorithms on both the public and real-world datasets. The code is available at https://github.com/ziiho08/Attention-LRCN.

Academic Emotion Classification Using FER: A Systematic Review

Facial emotion expressions are among the most potent, natural, and powerful means of human communication. Due to the COVID-19 pandemic, educational institutions worldwide are forced to switch rapidly to remote and online learning. Students are currently in an emergency state and must adapt to various and readily accessible learning methods, such as mobile learning applications or an e-learning system. A systematic literature review (SLR) is conducted to extract and synthesize information such as the emotion classifier used in the facial expression recognition (FER) system, the dataset used, the preprocessing technique applied, the feature extraction approach used, and the strength and limitation of the previous studies. Based on the search criteria, 701 publications were initially retrieved from five different digital databases, of which 48 studies have been chosen as primary studies for further analysis. Based on the findings of this study, the deep learning approach is the most frequently adopted approach in classifying student emotions during online learning. FER-2013 is the most commonly used FER dataset in FER studies, while DAiSEE is the most used academic emotion dataset. Moreover, support vector machine (SVM) is the conventional learning emotion classifier that is widely used in the FER systems, while convolutional neural network (CNN) is the most frequently used deep learning classifier. Next, it was found that the number of real-time FER systems is less than that of non-real-time FER systems. Finally, the top-1 accuracy of 94.6% was achieved by the long-term recurrent convolutional network on the academic emotion dataset, and the limitation is that it has low illumination and a lack of frontal pose.

Recognizing Human Activity Using Hybrid Models of CNN and LSTM in Deep Learning

Human Activity Recognition (HAR) is a time series categorization challenge that requires data from a number of timesteps in order to correctly classify the activities that are carried out. In recent times, the usage of image datasets for activity recognition has increased, however good classification cannot be done with just one frame. To increase recognition accuracy, multiple frames of data and the context of environment are required. It is known that a video is made up of a number of still images (frames) that are quickly updated to create the illusion of motion. The hybrid models of Deep Learning (DL) algorithms like Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) are proposed for recognising the human activity from video dataset. The hybrid models, Convolutional Long Short-Term Memory (ConvLSTM) and Long-term Recurrent Convolutional Network (LRCN) are introduced to improve the accuracy of HAR on video dataset. The models will be evaluated on standard video datasets, and the outcomes will show how HAR has the potential to significantly influence a number of industries. It has many applications in the fields of security, sports, and healthcare.

Operational solar flare forecasting via video-based deep learning

Operational flare forecasting aims at providing predictions that can be used to make decisions, typically on a daily scale, about the space weather impacts of flare occurrence. This study shows that video-based deep learning can be used for operational purposes when the training and validation sets used for network optimization are generated while accounting for the periodicity of the solar cycle. Specifically, this article describes an algorithm that can be applied to build up sets of active regions that are balanced according to the flare class rates associated to a specific cycle phase. These sets are used to train and validate a long-term recurrent convolutional network made of a combination of a convolutional neural network and a long short-term memory network. The reliability of this approach is assessed in the case of two prediction windows containing the solar storms of March 2015, June 2015, and September 2017.

Deep-Learning-Based Sequence Causal Long-Term Recurrent Convolutional Network for Data Fusion Using Video Data

The purpose of AI-Based schemes in intelligent systems is to advance and optimize system performance. Most intelligent systems adopt sequential data types derived from such systems. Realtime video data, for example, are continuously updated as a sequence to make necessary predictions for efficient system performance. The majority of deep-learning-based network architectures such as long short-term memory (LSTM), data fusion, two streams, and temporal convolutional network (TCN) for sequence data fusion are generally used to enhance robust system efficiency. In this paper, we propose a deep-learning-based neural network architecture for non-fix data that uses both a causal convolutional neural network (CNN) and a long-term recurrent convolutional network (LRCN). Causal CNNs and LRCNs use incorporated convolutional layers for feature extraction, so both architectures are capable of processing sequential data such as time series or video data that can be used in a variety of applications. Both architectures also have extracted features from the input sequence data to reduce the dimensionality of the data and capture the important information, and learn hierarchical representations for effective sequence processing tasks. We have also adopted a concept of series compact convolutional recurrent neural network (SCCRNN), which is a type of neural network architecture designed for processing sequential data combined by both convolutional and recurrent layers compactly, reducing the number of parameters and memory usage to maintain high accuracy. The architecture is challenge-able and suitable for continuously incoming sequence video data, and doing so allowed us to bring advantages to both LSTM-based networks and CNNbased networks. To verify this method, we evaluated it through a sequence learning model with network parameters and memory that are required in real environments based on the UCF-101 dataset, which is an action recognition data set of realistic action videos, collected from YouTube with 101 action categories. The results show that the proposed model in a sequence causal long-term recurrent convolutional network (SCLRCN) provides a performance improvement of at least 12% approximately or more to be compared with the existing models (LRCN and TCN).