Real-time Video Sequences Research Articles

AI-ENHANCED TRACKSEGNET AN ADVANCED MACHINE LEARNING TECHNIQUE FOR VIDEO SEGMENTATION AND OBJECT TRACKING

Video segmentation and object tracking are critical tasks in computer vision with applications spanning surveillance, autonomous driving, and interactive media. Traditional methods often struggle with the dynamic nature of video data, where object occlusions, variations in illumination, and complex motion patterns present significant challenges. Existing segmentation and tracking systems frequently suffer from inaccuracies in handling real-time video sequences, particularly in distinguishing and tracking multiple overlapping objects. The limitations of current models in addressing these issues necessitate the development of more advanced techniques that can effectively manage dynamic scenes and improve tracking accuracy. To address these challenges, we propose an advanced machine learning technique, AI-Enhanced TrackSegNet, which integrates deep learning with novel attention mechanisms for improved video segmentation and object tracking. Our method utilizes a combination of Convolutional Neural Networks (CNNs) for feature extraction and Long Short-Term Memory (LSTM) networks for temporal sequence modeling. We introduce an attention-based mechanism to dynamically focus on relevant features, enhancing the model's ability to handle occlusions and varying object appearances. The model was trained on a diverse dataset of video sequences, incorporating both synthetic and real-world footage. The AI-Enhanced TrackSegNet demonstrated significant improvements in performance compared to existing techniques. Our method achieved an average Intersection over Union (IoU) score of 86.7% for segmentation and a tracking precision rate of 91.3% on the MOT17 benchmark dataset. These results represent a 10.2% improvement in IoU and a 7.5% increase in tracking precision compared to state-of-the-art methods. The model also exhibited enhanced robustness in complex scenes, handling occlusions and motion variations with greater accuracy.

Contraction assessment of abdominal muscles using automated segmentation designed for wearable ultrasound applications

PurposeWearable ultrasound devices can be used to continuously monitor muscle activity. One possible application is to provide real-time feedback during physiotherapy, to show a patient whether an exercise is performed correctly. Algorithms which automatically analyze the data can be of importance to overcome the need for manual assessment and annotations and speed up evaluations especially when considering real-time video sequences. They even could be used to present feedback in an understandable manner to patients in a home-use scenario. The following work investigates three deep learning based segmentation approaches for abdominal muscles in ultrasound videos during a segmental stabilizing exercise. The segmentations are used to automatically classify the contraction state of the muscles.MethodsThe first approach employs a simple 2D network, while the remaining two integrate the time information from the videos either via additional tracking or directly into the network architecture. The contraction state is determined by comparing measures such as muscle thickness and center of mass between rest and exercise. A retrospective analysis is conducted but also a real-time scenario is simulated, where classification is performed during exercise.ResultsUsing the proposed segmentation algorithms, 71% of the muscle states are classified correctly in the retrospective analysis in comparison to 90% accuracy with manual reference segmentation. For the real-time approach the majority of given feedback during exercise is correct when the retrospective analysis had come to the correct result, too.ConclusionBoth retrospective and real-time analysis prove to be feasible. While no substantial differences between the algorithms were observed regarding classification, the networks incorporating the time information showed temporally more consistent segmentations. Limitations of the approaches as well as reasons for failing cases in segmentation, classification and real-time assessment are discussed and requirements regarding image quality and hardware design are derived.

Analysing the performance of Viola-Jones and multi-task convolution neural networks face detection algorithms using real-time video sequences

Analysing the performance of Viola-Jones and multi-task convolution neural networks face detection algorithms using real-time video sequences

Detecting abnormal behavior in megastore for intelligent surveillance through 3D deep convolutional model

Abstract The use of neural networks in a range of academic and scientific pursuits has introduced a great interest in modeling human behavior and activity patterns to recognize particular events. Various methods have so far been proposed for building expert vision systems to understand the scene and draw true semantic inferences from the observed dynamics. However, classifying abnormal or unusual activities in real-time video sequences is still challenging, as the details in video sequences have a time continuity constraint. A cost-effective approach is still demanding and so this work presents an advanced three-dimensional convolutional network (A3DConvNet) for detecting abnormal behavior of persons by analyzing their actions. The network proposed is 15 layers deep that uses 18 convolutional operations to effectively analyze the video contents and produces spatiotemporal features. The integrated dense layer uses these features for the efficient learning process and the softmax layer is used as the output layer for labeling the sequences. Additionally, we have created a dataset that carries video clips to represent abnormal behaviors of humans in megastores/shops, which is a consequent contribution of this paper. The dataset includes five complicated activities in the shops/megastores: normal, shoplifting, drinking, eating, and damaging. By analyzing human actions, the proposed algorithm produces an alert if anything like abnormalities is found. The extensive experiments performed on the synthesized dataset demonstrate the effectiveness of our method, with achieved accuracy of up to 90.90%.

Face Mask and Social Distance Monitoring via Computer Vision and Deployable System Architecture

The coronavirus (COVID-19) is a lethal virus causing a rapidly infectious disease throughout the globe. Spreading awareness, taking preventive measures, imposing strict restrictions on public gatherings, wearing facial masks, and maintaining safe social distancing have become crucial factors in keeping the virus at bay. Even though the world has spent a whole year preventing and curing the disease caused by the COVID-19 virus, the statistics show that the virus can cause an outbreak at any time on a large scale if thorough preventive measures are not maintained accordingly. To fight the spread of this virus, technologically developed systems have become very useful. However, the implementation of an automatic, robust, continuous, and lightweight monitoring system that can be efficiently deployed on an embedded device still has not become prevalent in the mass community. This paper aims to develop an automatic system to simultaneously detect social distance and face mask violation in real-time that has been deployed in an embedded system. A modified version of a convolutional neural network, the ResNet50 model, has been utilized to identify masked faces in people. You Only Look Once (YOLOv3) approach is applied for object detection and the DeepSORT technique is used to measure the social distance. The efficiency of the proposed model is tested on real-time video sequences taken from a video streaming source from an embedded system, Jetson Nano edge computing device, and smartphones, Android and iOS applications. Empirical results show that the implemented model can efficiently detect facial masks and social distance violations with acceptable accuracy and precision scores.

OGC P03 Staging of Barrett's neoplasia using artificial neural networks, proof of concept study

Abstract Background Endoscopic differentiation between intramucosal and submucosal Barrett's neoplasia has several important implications but remains challenging even for expert endoscopists. Recent studies demonstrated promising results on AI-assisted detection of Barrett's neoplasia, but data on AI-assisted staging is limited. We aimed to develop and validate an AI system for classification of Barrett's neoplasia into intramucosal or submucosal, and compare its performance to expert endoscopists. Methods The model, based on VGG-16 architecture, was trained on 117 images of prospectively collected and annotated Barrett's neoplastic lesions. Rotation and random flip were used for data augmentation. The ground truth was the histological staging of endoscopically resected specimens performed by two pathologists with expertise in Barrett's neoplasia. Images comprised of WLI, enhanced imaging, and magnification views. The model was designed to classify images as either intramucosal (pT1a) or submucosal (pT1b). Performance of the AI system was compared to a group of three experts. Results The AI model was tested on an independent dataset of 90 images. The accuracy, sensitivity and specificity and AUC of the AI model in differentiating between intramucosal and submucosal neoplasia was 70.9%, 72.5%, 65.7%, and 0.781 respectively. Mean accuracy, sensitivity and specificity of experts were 73.3%, 63.3% and 83.3% respectively. Processing speed of the AI system was 5 ms/image. Conclusions This study demonstrates the feasibility of AI-assisted staging of Barrett's neoplasia on endoscopic images. The AI model's performance was comparable to that of experts. More work is needed to further develop this early model and validate its use on real-time video sequences.

Development and validation of artificial neural networks model for detection of Barrett’s neoplasia: a multicenter pragmatic nonrandomized trial (with video)

The aim of this study was to develop and externally validate a computer-aided detection (CAD) system for the detection and localization of Barrett's neoplasia and assess its performance compared with that of general endoscopists in a statistically powered multicenter study by using real-time video sequences. In phase 1, the hybrid visual geometry group 16-SegNet model was trained by the use of 75,198 images and videos (96 patients) of neoplastic and 1,014,973 images and videos (65 patients) of nonneoplastic Barrett's esophagus. In phase 2, image-based validation was performed on a separate dataset of 107 images (20 patients) of neoplastic and 364 images (14 patients) of nonneoplastic Barrett's esophagus. In phase 3 (video-based external validation) we designed a real-time video-based study with 32 videos (32 patients) of neoplastic and 43 videos (43 patients) of nonneoplastic Barrett's esophagus from 4 European centers to compare the performance of the CAD model with that of 6 nonexpert endoscopists. The primary endpoint was the sensitivity of CAD diagnosis of Barrett's neoplasia. In phase 2, CAD detected Barrett's neoplasia with sensitivity, specificity, and accuracy of 95.3%, 94.5%, and 94.7%, respectively. In phase 3, the CAD system detected Barrett's neoplasia with sensitivity, specificity, negative predictive value, and accuracy of 93.8%, 90.7%, 95.1%, and 92.0%, respectively, compared with the endoscopists' performance of 63.5%, 77.9%, 74.2%, and 71.8%, respectively (P< .05 in all parameters). The CAD system localized neoplastic lesions with accuracy, mean precision, and mean intersection over union of 100%, 0.62, and 0.54, respectively, when compared with at least 1 of the expert markings. The processing speed of the CAD detection and localization were 5 ms/image and 33 ms/image, respectively. To our knowledge, this is the first study describing external (multicenter) validation of AI algorithms for the detection of Barrett's neoplasia on real-time endoscopic videos. The CAD system in this study significantly outperformed nonexpert endoscopists on real-time video-based assessment, achieving >90% sensitivity for neoplasia detection. This result needs to be validated during real-time endoscopic assessment.

Identification of crowd behaviour patterns using stability analysis

Crowd behaviour analysis and management have become a significant research problem for the last few years because of the substantial growth in the world population and their security requirements. There are numerous unsolved problems like crowd flow modelling and crowd behaviour detection, which are still open in this area, seeking great attention from the research community. Crowd flow modelling is one of such problems, and it is also an integral part of an intelligent surveillance system. Modelling of crowd flow has now become a vital concern in the development of intelligent surveillance systems. Real-time analysis of crowd behavior needs accurate models that represent crowded scenarios. An intelligent surveillance system supporting a good crowd flow model will help identify the risks in a wide range of emergencies and facilitate human safety. Mathematical models of crowd flow developed from real-time video sequences enable further analysis and decision making. A novel method identifying eight possible crowd flow behaviours commonly seen in the crowd video sequences is explained in this paper. The proposed method uses crowd flow localisation using the Gunnar-Farneback optical flow method. The Jacobian and Hessian matrix analysis along with corresponding eigenvalues helps to find stability points identifying the flow patterns. This work is carried out on 80 videos taken from UCF crowd and CUHK video datasets. Comparison with existing works from the literature proves our method yields better results.

Swarm-Based and Energy-Aware Unmanned Aerial Vehicle System for Video Delivery of Mobile Objects

Floods are the most common disaster in the world and cause deaths, damages to houses, buildings, and possessions, as well as disruption to communications. In such dynamic scenarios, it is needed to search and track mobile objects transported by the water flows, such as humans, animals, vehicles, and debris. The use of Unmanned Aerial Vehicles (UAVs) is essential in disaster scenarios to help first responders determine correct procedures in terms of searching, tracking, and rescuing the victims, as well as in defining the actions to minimize the risks in a sustainable and timely manner. However, the tracking of mobile objects and the delivery of real-time video sequences from UAVs with energy constraints toward first responders is still a hard task. Therefore, it is necessary to orchestrate a swarm of UAVs for searching and tracking objects while reacting fast to frequent changes in the topology and trajectory, as well as distributing real-time video flows with quality level support to the ground team. This article proposes an energy-aware swarm-based and mobility prediction scheme for UAVs, called SUAV. SUAV provides a unique UAV-based system for rescue scenarios with mobile objects and with support for route and path planning, searching and tracking procedures, mobility prediction and multi-hop communication management, video delivery with Quality of Service (QoS)/Quality of Experience (QoE) support, and energy-efficiency. Simulation results show the efficiency of SUAV in transmitting real-time video flows with QoS/QoE support while minimizing the energy consumption in flooding scenarios with mobile targets.

GSAS: Enhancing efficiency of human activity recognition using GRU based Sub-activity stitching

Recognition of human activities requires the design of highly efficient models for key-frame selection, image segmentation, feature extraction and selection, sub-activity classification, and sub-activity stitching. Due to the advent of effective frame selection and segmentation methods, most of the research in human activity recognition (HAR) is focused on feature extraction and its post-processive classification. Most of these models work on the principle of direct activity recognition, wherein the entire video sequence is scanned, and the final activity is recognized from it. In this approach has some lacunas, which include, limited accuracy due to classification of a large number of frames; limited scalability due to large delays for training and evaluation, and limited applicability due to the requirement of large datasets for model training and validation. To remove these drawbacks, this text proposes the design of a novel gated recurrent unit (GRU) based sub-activity stitching model (GSAS). The proposed model works in 2 phases, which include, sub-activity recognition using GRU, followed by sub-activity stitching using a 1D convolutional neural network (CNN). The former converts input frames into numerical sub-activity instances, while the latter processes these instances and converts them into final activity types. Due to the use of this 2-layered architecture, the proposed model can achieve a precision of 89%, a recall of 85%, and an accuracy of 90% which is higher than most of the recently proposed HAR models. The evaluation was done on untrimmed datasets, which makes the proposed model applicable to real-time video sequences. This text also recommends some transfer-learning approaches that can be applied to the proposed model to further improve its performance in terms of scalability and accuracy.

Visual-Feedback-Based Frame-by-Frame Synchronization for 3000 fps Projector–Camera Visual Light Communication

This paper proposes a novel method for synchronizing a high frame-rate (HFR) camera with an HFR projector, using a visual feedback-based synchronization algorithm for streaming video sequences in real time on a visible-light communication (VLC)-based system. The frame rates of the camera and projector are equal, and their phases are synchronized. A visual feedback-based synchronization algorithm is used to mitigate the complexities and stabilization issues of wire-based triggering for long-distance systems. The HFR projector projects a binary pattern modulated at 3000 fps. The HFR camera system operates at 3000 fps, which can capture and generate a delay signal to be given to the next camera clock cycle so that it matches the phase of the HFR projector. To test the synchronization performance, we used an HFR projector–camera-based VLC system in which the proposed synchronization algorithm provides maximum bandwidth utilization for the high-throughput transmission ability of the system and reduces data redundancy efficiently. The transmitter of the VLC system encodes the input video sequence into gray code, which is projected via high-definition multimedia interface streaming in the form of binary images 590 × 1060. At the receiver, a monochrome HFR camera can simultaneously capture and decode 12-bit 512 × 512 images in real time and reconstruct a color video sequence at 60 fps. The efficiency of the visual feedback-based synchronization algorithm is evaluated by streaming offline and live video sequences, using a VLC system with single and dual projectors, providing a multiple-projector-based system. The results show that the 3000 fps camera was successfully synchronized with a 3000 fps single-projector and a 1500 fps dual-projector system. It was confirmed that the synchronization algorithm can also be applied to VLC systems, autonomous vehicles, and surveillance applications.

ESTIMATING THE SURVIVAL OF AQUA FAUNA BY IMAGE ANALYSIS TECHNIQUES FROM THE VIDEO SEQUENCES OF POND AQUACULTURE

Survival status of aqua fauna is estimated by studying the dynamic behaviour (motion) in underwater environment. Survival status of aqua fauna particularly, shrimp is estimated to prevent huge economic losses to the aqua farmers. A novel approach has been proposed in this research work to address this issue through segmentation, extraction of features, motion analysis and classification of aqua fauna from the pond aquaculture video sequences. First, the content-based video is translated into frames, and the image content is segmented using the Background Subtraction algorithm. Then, shrimp motion analysis is done with the assistance of the optical Flow algorithm. At the same time, shrimp body feature is extracted by Edge detection method for Shape analysis. Finally, the Decision Tree classifier is constructed to estimate the status of shrimp is alive or dead. Real-time video sequences of shrimp in pond aquaculture have illustrated the effectiveness of our process. The projected method shows that the results are more accurate to estimate the shrimp survival status as well as providing GMP (Good Manufacturing Practices) in shrimp cultivation to alert the aqua farmers in advance to avoid economic loss

Study and implementation of K-multiple constraint shortest path for H.265 HEVC for optimal video compression

Today, everyone is demanding high-quality video services along with large bandwidth. Due to this demand, a ruthless burden is felt across the network architecture. Thus, a recently developed video codec H.265, also called high-efficiency video coding (HEVC), offers high coding efficiency than its predecessor H.264 Advanced Video Coding. Although H.265 or HEVC offers a greatly improved compression ratio, it is done at the expense of a high increase in computational complexity and time complexity to process the video compression. This disadvantage of HEVC makes it less suitable for real-time video sequences that may contain varying motion activities. Therefore, it is insufficient to use conventional motion estimation techniques that consume 80–90% of the total computational power. The research work presented in this paper deals with these two-short coming of H.265 HEVC. In this paper, first, an improved motion estimation technique is presented, which yields higher compression efficiency, computational complexities and better video quality, especially for low and medium video resolution applications such as handheld devices based videotelephony. Experimental results using improved motion estimation technique based on Quadtree decomposition and A* prune algorithm optimization reveals that total bits for different multi constraints shortest path get reduced while retaining the quality of video in reconstruction part. Besides, it can also be used in various multimedia applications with restricted computational resources.

Real Time Face Expression Recognition along with Balanced FER2013 Dataset using CycleGAN

Human face expression recognition is an active research area that has massive applications in medical field, crime investigation, marketing, online learning, automobile safety and video games. The first part of this research defines a deep neural network model-based framework for recognizing the seven main types of facial expression, which are found in all cultures. The proposed methodology involves four stages: (a) pre-processing the FER2013 dataset through relabeling to avoid misleading results and getting rid of non-face and non-frontal faces; (b) design of an efficient stable Cycle Generative Adversarial Network (CycleGAN), which provides unsupervised expression-to-expression translation. The CycleGAN has been designed and trained with a new cycle consistency loss. (c) Generating new images to overcome the class imbalance and finally (d) building the DNN architecture for recognizing the face sign expression, using the pretrained VGG-Face model with vggface weights. The second part encompasses the design of a GPU-accelerated face expression recognition system for real time video sequences using NVIDIAs Compute Unified Device Architecture (CUDA). OpenCV library has been compiled from scratch with CUDA and NVIDIA CUDA Deep Neural Network library, cuDNN. For face detection stage Haar Cascaded and deep learning were used and tested using both CPU and GPU as backend. Results show that the designed model run time to recognize a face sign is 0.44 seconds. Besides, the average test accuracy has been increased from 64% for the original FER2013 dataset to 91.76% for the modified balanced version using the same transfer learning model.

Eye Tracking and ROI Detection within a Computer Screen Using a Monocular Camera

The region of interest will change according to the task, even in the same situation. In the study, a method for region of interest detection within a computer screen using a monocular camera is proposed. In contrast to gaze tracking techniques that require particular devices (e.g., an eye tracker and RGB-D device) meanwhile complex calibration, a cheap and more convenient monocular camera is used in this study to solves the eye gaze tracking problem. Firstly, Human face is detected in a real-time video sequence using HoG features. Then, the landmarks around the eyes, which reflect the gaze position, are extracted. Next, the iris centers are detected in the eye region. In order to reduce the gaze error caused by head movement, a three-dimensional head model is proposed to estimate head pose. Finally, the eye region is tracked by calculating the eye vectors and head movement. Experiments were performed to evaluate the face detection, landmarks, iris detection, eye movement estimation, and head pose estimation on databases such as the Hong Kong, BioID, and Boston University head pose databases. Besides, experiments for gaze tracking were performed for a real-time video sequence. Deviation is calculated using Euclidean distance between the real and estimated points. The results show that the method achieves an average error of 1.85∘ with head fixed and 3.58∘ with head movement in the range of −45∘ and 45∘. The requirement is detecting the user’s attention in the screen area. Our method can reach the same level to the other methods, even though the accuracy is not state-of-the-art. Meanwhile, as we all know not only a specific point is concerned but also a region area according to the characteristics of human eye imaging, thus the proposed method can meet the requirements of demand.

Human detection techniques for real time surveillance: a comprehensive survey

Real-time detection of humans is an evolutionary research topic. It is an essential and prominent component of various vision based applications. Detection of humans in real-time video sequences is an arduous and challenging task due to various constraints like cluttered environment, occlusion, noise, etc. Many researchers are doing their research in this area and have published the number of researches so far. Determining humans in visual monitoring system is prominent for different types of applications like person detection and identification, fall detection for an elder person, abnormal surveillance, gender classification, crowd analysis, person gait characterization, etc. The main objective of this paper is to provide a comprehensive survey of the various challenges and modern developments seen for human detection methodologies in day vision. This paper consists of an overview of different human detection techniques and their classification based on various underlying factors. The algorithmic technicalities with their applicability to these techniques are deliberated in detail in the manuscript. Different humanitarian imperative factors have also been highlighted for comparative analysis of each human detection methodology. Our survey shows the difference between current research and future requirements.

Human Crowd Detection for City Wide Surveillance

Abstract Surveillance systems are most commonly used for monitoring/surveillance of almost all public and private places. Real time behaviour of these systems add extra capacity to the surveillance activity. In such cases, any suspicious or uneven activity is detected by analysing the real time video stream of the cameras through these system. At maximum places, the surveillance activity is more manual, i.e. real time video footage of CCTV is monitored by security personals in some control room. It is difficult for any human being to monitor CCTV continuously and tirelessly without any rest. Therefore, robustness and effectiveness of such surveillance depends on technological strength of security personals, (i.e. human being), and hence pose the need of more man power for round the clock surveillance. This paper discusses a complete autonomous solution that will not only reduce the effort of individuals but will also increase the effectiveness of the law enforcement authority in patrolling the public places of a city. This system is proposed to use the existing CCTV infrastructure of the city. Various computer vision techniques are employed to detect the suspicious activity from real time video sequences. An intricate communication system is also designed to decrease the response time of system for the police authorities. This system is capable to generate an alarm signal if there founds any suspicious fellow or unusual activity in the restricted site. It also generates the complete description related to the events happenings in form of textual warning messages.

An improved dark channel prior based defogging algorithm for video sequences

Bad climatic situations comprising of haze, dimness, and residue frequently decrease the general execution of outside cameras. Fog carries an issue to PC vision/photographs applications as it drops deceivability of the scene. The murkiness is framed because of principal marvels, which may be a constriction and air-light. Weakening diminishes the differentiation and air-light will expand the whiteness inside the scene. Cloudiness disposal systems improve the shading and correlation of the stage. To enhance the adequacy of surveillance and in-vehicle cameras underneath such conditions, a method dependent on the dull channel earlier for snappy defogging of pictures is proposed. On this work to defog an image, dark channel prior based algorithm is carried out the use of atmospheric scattering version. The proposed algorithm estimates the air light-based totally on the input image, calculating the preliminary transmission rate map, refining it and then finally restoring the fog image and it is applied for a real-time video sequence, and the video frames are defogged.

Computer Vision based System to Detect Abandoned Objects

Security has become the most essential aspect in every walk of life. With ever growing technology, there has been an apparent inclination towards developing smart surveillance systems that do not stop with just monitoring and recording of events, but also possesses the ability to observe the events and alert in case of any discrepancies if observed. One of the impending threats to security in crowded places is the ignorance of un-attended objects. These un-attended objects or abandoned objects may have been left behind intentionally and may contain hazardous things which can cause huge disasters. This paper is focused towards developing a computer vision based approach that analyses the blob areas to detect any abandoned objects and instantaneously send appropriate alert without any human intervention. The uniqueness of this approach is that it handles the occlusion scenario and also quick execution time to detect abandoned objects. The approach has been tested with various benchmark video datasets and real-time video sequences as well. The performance has been measured in terms of accuracy of classifying abandoned objects in a given video sequence and also the execution time taken for computing the outcome. Results indicate good accuracy in terms of abandoned object detection under varying conditions and scenarios and also faster execution time when compared to other contemporary approaches.

Abnormality Detection Using LBP Features and K-Means Labelling based Feed-Forward Neural Network in Video Sequence

Video surveillance is widely used in various domains like military, commercial and consumer areas. One of the objectives in video surveillance is the detection of normal and abnormal behavior.It has always been a challenge to accurately identify such events in any real time video sequence. In this paper, abnormality detection method using Local Binary Pattern and k-means labeling basedfeed-forward neural network has been proposed. The performance of the proposed method has also been compared with four other techniques in literature to show its worthiness. It can be seen in the experimental results that an accuracy of up to 98% has been achieved for the proposed technique.