Real-time Video Research Articles

Revolutionizing Deepfake Detection and Realtime Video Vision with CNN-based Deep Learning Model

The rapid advancement of deep learning models that can generate and synthesis hyper-realistic videos known as Deepfakes and their ease of access have raised concern on possible malicious intent use. Deep learning techniques can now generate faces, swap faces between two subjects in a video, alter facial expressions, change gender, and alter facial features, to list a few. These powerful video manipulation methods have potential use in many fields. However, they also pose a looming threat to everyone if used for harmful purposes such as identity theft, phishing, and scam. In this work, we propose a Convolutional Vision Transformer for the detection of Deepfakes. The Convolutional Vision Transformer: Convolutional Neural Network (CNN) and Vision Transformer (ViT). The CNN extracts learnable features while the ViT takes in the learned features as input and categorizes them using an attention mechanism. We trained our model on the DeepFake Detection Challenge Dataset (DFDC) and have achieved 91.5 percent accuracy, an AUC value of 0.91, and a loss value of 0.32. Our contribution is that we have added a CNN module to the ViT architecture and have achieved a competitive result on the DFDC dataset.

Nerambum: A Virtual Exhibition Platform

In the wake of the global pandemic, industries worldwide faced a disruptive challenge, particularly those thriving on physical exhibitions essential for expanding market share, unveiling new products, and forging valuable connections. The health crisis and consequent strict guidelines rendered traditional exhibitions untenable, causing a catastrophic loss of audience and opportunities for businesses. This unprecedented situation accelerated the adoption of virtual alternatives, but existing platforms failed to capture the essence and dynamism of live, in-person interactions, leading to a palpable gap in user experience. "Nerambum" emerges as a transformative solution in this landscape, offering a sophisticated online exhibition platform that recreates the authentic feel of physical events. It seamlessly integrates a 3D environment for attendees with a multifunctional web application, supporting various user roles including admin, exhibition owners, stall owners, and general participants. Beyond its immersive virtual space, "Nerambum" stands out with its comprehensive service management—from secure authentication, customizable avatars, and stalls to real-time video streaming and a practical payment gateway mimicking physical ticket purchases. The platform not only adapts to the current necessity for remote interaction but also anticipates the enduring relevance of virtual engagements in the exhibition industry's future.

Integration of eye-tracking and object detection in a deep learning system for quality inspection analysis

Abstract During quality inspection in manufacturing, the gaze of a worker provides pivotal information for identifying surface defects of a product. However, it is challenging to digitize the gaze information of workers in a dynamic environment where the positions and postures of the products and workers are not fixed. A robust, deep learning-based system, ISGOD (Integrated System with worker’s Gaze and Object Detection), is proposed, which analyzes data to determine which part of the object is observed by integrating object detection and eye-tracking information in dynamic environments. The ISGOD employs a six-dimensional pose estimation algorithm for object detection, considering the location, orientation, and rotation of the object. Eye-tracking data were obtained from Tobii Glasses, which enable real-time video transmission and eye-movement tracking. A latency reduction method is proposed to overcome the time delays between object detection and eye-tracking information. Three evaluation indices, namely, gaze score, accuracy score, and concentration index are suggested for comprehensive analysis. Two experiments were conducted: a robustness test to confirm the suitability for real-time object detection and eye-tracking, and a trend test to analyze the difference in gaze movement between experts and novices. In the future, the proposed method and system can transfer the expertise of experts to enhance defect detection efficiency significantly.

PASS: Patch Automatic Skip Scheme for Efficient On-Device Video Perception.

Real-time video perception tasks are often challenging on resource-constrained edge devices due to the issues of accuracy drop and hardware overhead, where saving computations is the key to performance improvement. Existing methods either rely on domain-specific neural chips or priorly searched models, which require specialized optimization according to different task properties. These limitations motivate us to design a general and task-independent methodology, called Patch Automatic Skip Scheme (PASS), which supports diverse video perception settings by decoupling acceleration and tasks. The gist is to capture inter-frame correlations and skip redundant computations at patch level, where the patch is a non-overlapping square block in visual. PASS equips each convolution layer with a learnable gate to selectively determine which patches could be safely skipped without degrading model accuracy. Specifically, we are the first to construct a self-supervisory procedure for gate optimization, which learns to extract contrastive representations from frame sequences. The pre-trained gates can serve as plug-and-play modules to implement patch-skippable neural backbones, and automatically generate proper skip strategy to accelerate different video-based downstream tasks, e.g., outperforming state-of-the-art MobileHumanPose in 3D pose estimation and FairMOT in multiple object tracking, by up to 9.43 × and 12.19 × speedups, respectively, on NVIDIA Jetson Nano devices.

Real Time Weapon Detection using YOLOv8 and Alert Mechanism

Abstract: Security cameras and video surveillance cameras have become an important partof public safety. However, in many cities, thesesystems still manually detect high-risk situations. Understaffing in security services can lead to delays in detecting incidents or unforeseen threats, putting the public at risk. The aim of this project is to develop a low-cost, effective intelligencebased solution for real-time weapons detection and surveillance video analysis in different situations. As can be seen from many statistics, the incidence of gun and dangerous weapon crimes is increasing every year, making it difficult for the police to solve the problem in time. Crimes caused by guns or knives are very common in many places, especially in places where gun laws do not exist. Early detection of crime is critical to public safety. One way to prevent these situations is to use video surveillance to detect the presence of dangerous weapons such as guns and knives. Monitoring and control now also require monitoring and intervention. Here, we present in video a system for tracking weapons suitable for tracking and controlling targets. We use the YOLOv8 (look once) algorithm to detect weapons in live video. YOLO model is an end-to-end deep learning model; it is very popular because it is fast and accurate. Previous methods such as region-based convolutional neural networks (R-CNN) required thousands of network tests to make predictions for an image, which could be time-consuming-Optimization is a laborious and painful process. He focuses on specific areas ofpainting and trains everything personally. The YOLO model, on the other hand, passes the image through the neural network only once. Since speed isimportant in real-time video, we use the YOLOv8 algorithm. This data was trained to classify three groups of weapons (pistols, knives, and artillery). When a weapon is detected, an alert is sent to authorities who can take action and reduce crime before it becomes a crime. Powered by Tensorflow, the system took 294 seconds in the first test toidentify weapon types in 9 categories.

Real Time Weapon Detection using YOLOv8 and Alert Mechanism

Abstract: Security cameras and video surveillance cameras have become an important partof public safety. However, in many cities, thesesystems still manually detect high-risk situations. Understaffing in security services can lead to delays in detecting incidents or unforeseen threats, putting the public at risk. The aim of this project is to develop a low-cost, effective intelligencebased solution for real-time weapons detection and surveillance video analysis in different situations. As can be seen from many statistics, the incidence of gun and dangerous weapon crimes is increasing every year, making it difficult for the police to solve the problem in time. Crimes caused by guns or knives are very common in many places, especially in places where gun laws do not exist. Early detection of crime is critical to public safety. One way to prevent these situations is to use video surveillance to detect the presence of dangerous weapons such as guns and knives. Monitoring and control now also require monitoring and intervention.

Enhancing Object Detection Accuracy Through Custom Dataset Using Yolo

Abstract: Potholes pose significant risks to road safety and vehicle maintenance, leading to accidents and costly repairs. Traditional methods of pothole detection are often labour-intensive and time-consuming. In this study, we propose an innovative approach to pothole detection using YOLOv8, a state-of-the-art object detection algorithm. By harnessing the power of deep learning, our system can accurately identify and locate potholes in real-time video streams from traffic cameras and vehicles. We employ YOLOv8, an advanced variant of the You Only Look Once (YOLO) algorithm, known for its speed and accuracy in real-time object detection tasks. Leveraging a large annotated dataset of road images, we fine-tune the YOLOv8 model to specifically detect potholes. Our trained model is capable of identifying various pothole sizes and shapes, even in challenging lighting and weather conditions.

A Comparative Analysis of the Effects of Real-time Video Conference and Metaverse Online Classroom Platform on Learner Presence, Engagement, and Achievement: A Focus on Gathertown and Zoom

A Comparative Analysis of the Effects of Real-time Video Conference and Metaverse Online Classroom Platform on Learner Presence, Engagement, and Achievement: A Focus on Gathertown and Zoom

Connected Robotics for Intelligent Surveillance: An IOT Approach

Abstract: The robot employs the HTTP Communication protocol, enabling users to access live video streaming from the OV2640 camera through a standard web browser. The web interface includes intuitive directional buttons for seamless control, allowing users to command the robot to move left, right, forward, and backward. The locomotion of the robot is achieved through two DC motors integrated with a Robot chassis and an L293D motor driver module. This paper presents the development and implementation of an innovative IoT-based Surveillance Robot designed for remote control, utilizing a Wi-Fi module and a smartphone interface. The portable espionage robot integrates cutting-edge technology, featuring an AI Thinker ESP32-CAM module with an ESP32-S processor, an OV2640 camera, motor drivers, and a robust chassis with two moving wheels. The ESP32-CAM allows the capture and storage of images on a microSD card, facilitating efficient data management. The real-time video feed is transmitted wirelessly to the user's smartphone over a Wi-Fi infrastructure, providing a livestreaming experience. This wireless surveillance robot finds applications in various scenarios such as war fields, disaster management, and beyond, showcasing its versatility in remote monitoring and reconnaissance tasks.

Customer Interest in E-Commerce Live Streaming for Fashion Thrift Repurchases Systematic Literature Review

The rapid advancement of technology has revolutionized various aspects of human life, including online shopping and information exchange via the internet. One method to do online shopping is through e-commerce. The online shopping landscape continues to evolve with the emergence of E-Commerce Live Streaming (ECLS) as a new and interactive platform for consumer engagement. This research explores the utilization of ECLS as a medium to repurchase thrift fashion, a trend driven by sustainable fashion preferences and digital retail innovation. This research analyzes the factors influencing consumer repurchase intention in the thrift fashion industry to support local clothing sellers. ECLS combines real-time video broadcasting with interactive features, creating an immersive shopping experience that goes beyond traditional online retail. Researchers selected 30 articles that were most relevant to the topic. We looked at the key drivers shaping this intersection by adopting a multidisciplinary approach, including a literature review and qualitative analysis. From the appeal of personalized advice to the excitement of witnessing the appearance of products in real-time, consumers are attracted to the unique advantages that modern technology offers in the context of sustainable fashion choices. The results show that affordable prices, good quality of goods, and the uniqueness and diversity of products are the main factors that influence repurchase interest in thrift stores. In addition, environmental awareness and sentimental value of second-hand goods also contribute significantly to repurchase intention.

Multi-instance learning based artificial intelligence model to assist vocal fold leukoplakia diagnosis: A multicentre diagnostic study

ObjectiveTo develop a multi-instance learning (MIL) based artificial intelligence (AI)-assisted diagnosis models by using laryngoscopic images to differentiate benign and malignant vocal fold leukoplakia (VFL). MethodsThe AI system was developed, trained and validated on 5362 images of 551 patients from three hospitals. Automated regions of interest (ROI) segmentation algorithm was utilized to construct image-level features. MIL was used to fusion image level results to patient level features, then the extracted features were modeled by seven machine learning algorithms. Finally, we evaluated the image level and patient level results. Additionally, 50 videos of VFL were prospectively gathered to assess the system's real-time diagnostic capabilities. A human-machine comparison database was also constructed to compare the diagnostic performance of otolaryngologists with and without AI assistance. ResultsIn internal and external validation sets, the maximum area under the curve (AUC) for image level segmentation models was 0.775 (95 % CI 0.740–0.811) and 0.720 (95 % CI 0.684–0.756), respectively. Utilizing a MIL-based fusion strategy, the AUC at the patient level increased to 0.869 (95 % CI 0.798–0.940) and 0.851 (95 % CI 0.756–0.945). For real-time video diagnosis, the maximum AUC at the patient level reached 0.850 (95 % CI, 0.743–0.957). With AI assistance, the AUC improved from 0.720 (95 % CI 0.682–0.755) to 0.808 (95 % CI 0.775–0.839) for senior otolaryngologists and from 0.647 (95 % CI 0.608–0.686) to 0.807 (95 % CI 0.773–0.837) for junior otolaryngologists. ConclusionsThe MIL based AI-assisted diagnosis system can significantly improve the diagnostic performance of otolaryngologists for VFL and help to make proper clinical decisions.

Converged learning: the spectrum of technology-mediated learning

New Jersey Institute of Technology (NJIT), a four-year polytechnic research university in the United States, utilized a participatory strategic planning process to implement an innovative approach to the delivery modes between face-to-face and online instruction. NJIT defines the spectrum of online and on-ground instruction integration as Converged Education. This spectrum allows students to participate face-to-face, join remotely through real-time video conferencing technology (synchronously), or watch classroom instruction asynchronously. The article opens with a general background of NJIT's approach to the new idea of converged instructional delivery. Then, the process of defining and clarifying the terms and conceptions of the various modes to be adopted is presented. Finally, the resulting implementation of the new policy and its reflection on course offerings is shown and discussed.

A Neuro-Driven Mobile Robot Enhanced by LoRa and EEG Technology

The Cognitive Rover project presents an imaginative way to deal with portable mechanical technology, coordinating high level neuro-innovations with long-range correspondence capacities to make a flexible and versatile mechanical stage. The essential target of the task is to foster a versatile robot fit for independent route and connection with its current circumstance, directed by brain signals got through electroencephalography (EEG) innovation. Utilizing the LoRa (Long Range) correspondence convention, the robot lays out strong network overstretched distances, empowering far off activity and information transmission in different situations, including search and salvage missions, perilous climate investigation, and far off review errands. The proposed framework design comprises of two fundamental parts: a neuro-driven control framework and a correspondence module. The neuro-driven control framework uses EEG signals gathered from a MindWave headset to decipher client orders and natural boosts. Through AI calculations and continuous sign handling strategies, the framework makes an interpretation of EEG signals into noteworthy orders for independent route, obstruction evasion, and undertaking execution. The correspondence module incorporates LoRa innovation to work with consistent information trade between the robot and far off administrators, empowering continuous observing, control, and video real time over significant distances. Key benefits of the Cognitive Rover incorporate its versatility to assorted conditions, instinctive client association through cerebrum PC interface innovation, and vigorous long-range correspondence abilities. Additionally, the task adds to progressions in mechanical technology, human-robot cooperation, and assistive advancements, with expected applications in misfortune reaction, ecological checking, and foundation review. Key Words: Electroencephalography (EEG), LoRa (Long Range), Cognitive Rover (CR), Brain Computer Interface (BCI), Bluetooth Low Energy (BLE), Transmitter (Tx)

Real-Time Multi-Person Video Synthesis with Controllable Prior-Guided Matting.

In order to enhance the matting performance in multi-person dynamic scenarios, we introduce a robust, real-time, high-resolution, and controllable human video matting method that achieves state of the art on all metrics. Unlike most existing methods that perform video matting frame by frame as independent images, we design a unified architecture using a controllable generation model to solve the problem of the lack of overall semantic information in multi-person video. Our method, called ControlMatting, uses an independent recurrent architecture to exploit temporal information in videos and achieves significant improvements in temporal coherence and detailed matting quality. ControlMatting adopts a mixed training strategy comprised of matting and a semantic segmentation dataset, which effectively improves the semantic understanding ability of the model. Furthermore, we propose a novel deep learning-based image filter algorithm that enforces our detailed augmentation ability on both matting and segmentation objectives. Our experiments have proved that prior information about the human body from the image itself can effectively combat the defect masking problem caused by complex dynamic scenarios with multiple people.

Performance analysis of collaborative real-time video quality of service prediction with machine learning algorithms

Performance analysis of collaborative real-time video quality of service prediction with machine learning algorithms

Edge computing based real-time Nephrops (Nephrops norvegicus) catch estimation in demersal trawls using object detection models

In demersal trawl fisheries, the unavailability of the catch information until the end of the catching process is a drawback, leading to seabed impacts, bycatches and reducing the economic performance of the fisheries. The emergence of in-trawl cameras to observe catches in real-time can provide such information. This data needs to be processed in real-time to determine the catch compositions and rates, eventually improving sustainability and economic performance of the fisheries. In this study, a real-time underwater video processing system counting the Nephrops individuals entering the trawl has been developed using object detection and tracking methods on an edge device (NVIDIA Jetson AGX Orin). Seven state-of-the-art YOLO models were tested to discover the appropriate training settings and YOLO model. To achieve real-time processing and accurate counting simultaneously, four frame skipping ideas were evaluated. It has been shown that adaptive frame skipping approach, together with YOLOv8s model, can increase the processing speed up to 97.47 FPS while achieving correct count rate and F-score of 82.57% and 0.86, respectively. In conclusion, this system can improve the sustainability of the Nephrops directed trawl fishery by providing catch information in real-time.

Automated Upper Tract Urothelial Carcinoma Tumor Segmentation During Ureteroscopy Using Computer Vision Techniques.

Introduction: Endoscopic tumor ablation of upper tract urothelial carcinoma (UTUC) allows for tumor control with the benefit of renal preservation but is impacted by intraoperative visibility. We sought to develop a computer vision model for real-time, automated segmentation of UTUC tumors to augment visualization during treatment. Materials and Methods: We collected 20 videos of endoscopic treatment of UTUC from two institutions. Frames from each video (N = 3387) were extracted and manually annotated to identify tumors and areas of ablated tumor. Three established computer vision models (U-Net, U-Net++, and UNext) were trained using these annotated frames and compared. Eighty percent of the data was used to train the models while 10% was used for both validation and testing. We evaluated the highest performing model for tumor and ablated tissue segmentation using a pixel-based analysis. The model and a video overlay depicting tumor segmentation were further evaluated intraoperatively. Results: All 20 videos (mean 36 ± 58 seconds) demonstrated tumor identification and 12 depicted areas of ablated tumor. The U-Net model demonstrated the best performance for segmentation of both tumors (area under the receiver operating curve [AUC-ROC] of 0.96) and areas of ablated tumor (AUC-ROC of 0.90). In addition, we implemented a working system to process real-time video feeds and overlay model predictions intraoperatively. The model was able to annotate new videos at 15 frames per second. Conclusions: Computer vision models demonstrate excellent real-time performance for automated upper tract urothelial tumor segmentation during ureteroscopy.

A comprehensive analysis of real-time video anomaly detection methods for human and vehicular movement

A comprehensive analysis of real-time video anomaly detection methods for human and vehicular movement

Man and the Machine: Effects of AI-assisted Human Labeling on Interactive Annotation of Real-time Video Streams

AI-assisted interactive annotation is a powerful way to facilitate data annotation—a prerequisite for constructing robust AI models. While AI-assisted interactive annotation has been extensively studied in static settings, less is known about its usage in dynamic scenarios where the annotators operate under time and cognitive constraints, e.g., while detecting suspicious or dangerous activities from real-time surveillance feeds. Understanding how AI can assist annotators in these tasks and facilitate consistent annotation is paramount to ensure high performance for AI models trained on these data. We address this gap in interactive machine learning (IML) research, contributing an extensive investigation of the benefits, limitations, and challenges of AI-assisted annotation in dynamic application use cases. We address both the effects of AI on annotators and the effects of (AI) annotations on the performance of AI models trained on annotated data in real-time video annotations. We conduct extensive experiments that compare annotation performance at two annotator levels (expert and non-expert) and two interactive labeling techniques (with and without AI assistance). In a controlled study with \(N=34\) annotators and a follow-up study with 51,963 images and their annotation labels being input to the AI model, we demonstrate that the benefits of AI-assisted models are greatest for non-expert users and for cases where targets are only partially or briefly visible. The expert users tend to outperform or achieve similar performance as the AI model. Labels combining AI and expert annotations result in the best overall performance as the AI reduces overflow and latency in the expert annotations. We derive guidelines for the use of AI-assisted human annotation in real-time dynamic use cases.

Real-time identification and visualization of Egyptian blue using modified night vision goggles

The possibility to use light in the visible spectrum to induce near-infrared luminescence in some materials, particularly Egyptian blue and related pigments, offers a significant advantage in terms of their detection. Since 2008, this property has been exploited to reveal the presence of those pigments even in tiny amounts on ancient and decayed surfaces, using a technical-photography method. This paper presents a new type of imaging device that enables real-time, easy, and inexpensive identification and mapping of Egyptian blue and related materials. The potential of the new tool is demonstrated by its effectiveness in detecting Egyptian blue within some prestigious sites: (a) Egyptian findings at Museo Egizio, Turin; (b) underground Roman frescoes at Domus Aurea, Rome; and (c) Renaissance frescoes by Raphael, Triumph of Galatea and Loggia of Cupid and Psyche, at Villa Farnesina, Rome. The device is based on night vision technology and allows an unprecedented fast, versatile, and user-friendly approach. It is employable by professionals including archeologists, conservators, and conservation scientists, as well as by untrained individuals such as students or tourists at museums and sites. The overall aim is not to replace existing photographic techniques but to develop a tool that enables rapid preliminary recognition, useful for planning the work to be carried out with conventional methods. The ability to immediately track Egyptian blue and related pigments, through real-time vision, photos, and videos, also provides a new kind of immersive experience (Blue Vision) and can foster the modern use of these materials in innovative applications and future technologies.