Real-time Video Research Articles

Integrating Depth-Based and Deep Learning Techniques for Real-Time Video Matting without Green Screens

Virtual production, a filmmaking technique that seamlessly merges virtual and real cinematography, has revolutionized the film and television industry. However, traditional virtual production requires the setup of green screens, which can be both costly and cumbersome. We have developed a green screen-free virtual production system that incorporates a 3D tracker for camera tracking, enabling the compositing of virtual and real-world images from a moving camera with varying perspectives. To address the core issue of video matting in virtual production, we introduce a novel Boundary-Selective Fusion (BSF) technique that combines the alpha mattes generated by deep learning-based and depth-based approaches, leveraging their complementary strengths. Experimental results demonstrate that this combined alpha matte is more accurate and robust than those produced by either method alone. Overall, the proposed BSF technique is competitive with state-of-the-art video matting methods, particularly in scenarios involving humans holding objects or other complex settings. The proposed system enables real-time previewing of composite footage during filmmaking, reducing the costs associated with green screen setups and simplifying the compositing process of virtual and real images.

Data safety prediction using YOLOv7+G3HN for traffic roads

Pulau Pinang has introduced several measures to enhance traffic safety and promote sustainability, including the installation of CCTV systems and the implementation of smart solutions and green technology as part of the Penang 2030 vision, aligning with the Sustainable Development Goals (SDGs). However, despite these efforts, road accidents persist due to non-optimised detection models, incomplete data from manual reporting, and technological constraints in real-time video analysis and predictive modelling. This study evaluates the effectiveness of the YOLOv7+G3HN framework for vehicle detection and near-miss analysis, with a focus on the influence of video quality on detection performance. The research aims to understand how high- and low-quality video inputs affect the accuracy and computational efficiency of detection algorithms. High[1]quality videos resulted in significantly faster computation times for vehicle detection than low-quality videos, highlighting the importance of video resolution in optimising detection processes. Despite the robustness of the algorithm, with no errors detected in both video qualities, higher miss detection rates in low-quality videos suggest that lower resolution may compromise detection accuracy and the effectiveness of monitoring systems. Near-miss analysis revealed that high-quality videos had a lower probability of near-miss occurrences than low-quality videos, highlighting the importance of video resolution for detection efficacy. These findings emphasise the critical role of high-resolution video inputs in enhancing detection accuracy and reliability, advocating for their implementation to optimise vehicle detection and improve road safety. Additionally, YOLOv7+G3HN outperforms YOLOv7 in both accuracy and speed. The study concludes that the YOLOv7+G3HN framework is effective for vehicle detection and near-miss analysis, provided that video quality is considered in system design and implementation.

Analysis of Convolutional Neural Network-based Image Classifications: A Multi-Featured Application for Rice Leaf Disease Prediction and Recommendations for Farmers

This study presents a novel method for improving rice disease classification using 8 different convolutional neural network (CNN) algorithms, which will further the field of precision agriculture. A thorough investigation of deep learning methods is carried out using the UCI dataset in order to create a reliable and effective model that can correctly identify a range of rice diseases. The suggested transfer learning models performs better at identifying subtle features and complex patterns in the dataset, which results in extremely accurate disease classification. Moreover, the study goes beyond the creation of models by incorporating an intuitive Tkinter-based application that offers farmers a feature-rich interface. With the help of this cutting-edge application, farmers will be able to make timely and well-informed decisions by enabling real-time disease prediction and providing personalized recommendations. Together with the user-friendly Tkinter interface, the smooth integration of cutting-edge CNN transfer Learning algorithms-based technology that include ResNet-50, InceptionV3, VGG16, MobileNetv2 with the UCI dataset represents a major advancement toward modernizing agricultural practices and guaranteeing sustainable crop management. Remarkable outcomes include 75% accuracy for ResNet-50, 90% accuracy for DenseNet121, 84% accuracy for VGG16, 95.83% accuracy for MobileNetV2, 91.61% accuracy for DenseNet169, and 86% accuracy for InceptionV3. These results give a concise summary of the models' capabilities, assisting researchers in choosing appropriate strategies for precise and successful rice crop disease identification. A severe overfitting has been seen on VGG19 with 70% accuracy and Nasnet with 80.02% accuracy. On Renset101 only an accuracy of 54% could be achieved along with only 33% on efficientNetB0. MobileNetV2 trained model was successfully deployed on a tkinter GUI application to make predictions using image or real time video capturing.

Operational Tests for Delay-Tolerant Network between the Moon and Earth Using the Korea Pathfinder Lunar Orbiter in Lunar Orbit

The Korea Pathfinder Lunar Orbiter (KPLO) was launched on 5 August 2022, equipped on the SpaceX Falcon 9 launch vehicle. At present, the KPLO is effectively carrying out its scientific mission in lunar orbit. The KPLO serves as a cornerstone for the development and validation of Korean space science and deep space technology. Among its payloads is the DTNPL, enabling the first-ever test of delay-tolerant network (DTN) technology for satellites in lunar orbit. DTN technology represents a significant advancement in space communication, offering stable communication capabilities characterized by high delay tolerance, reliability, and asymmetric communication speeds—a necessity for existing satellite and space communication systems to evolve. In this paper, we briefly give an overview of the Korea Lunar Exploration Program (KLEP) and present scientific data gathered through the KPLO mission. Specifically, we focus on the operational tests for DTN-ION conducted for message and file transfer, as well as real-time video streaming, during the initial operations of the KPLO. Lastly, this study offers insights and lessons learned from KPLO DTNPL operations, with the goal of providing valuable guidance for future advancements in space communication.

Robotic scrub nurse to anticipate surgical instruments based on real-time laparoscopic video analysis

BackgroundMachine learning and robotics technologies are increasingly being used in the healthcare domain to improve the quality and efficiency of surgeries and to address challenges such as staff shortages. Robotic scrub nurses in particular offer great potential to address staff shortages by assuming nursing tasks such as the handover of surgical instruments.MethodsWe introduce a robotic scrub nurse system designed to enhance the quality of surgeries and efficiency of surgical workflows by predicting and delivering the required surgical instruments based on real-time laparoscopic video analysis. We propose a three-stage deep learning architecture consisting of a single frame-, temporal multi frame-, and informed model to anticipate surgical instruments. The anticipation model was trained on a total of 62 laparoscopic cholecystectomies.ResultsHere, we show that our prediction system can accurately anticipate 71.54% of the surgical instruments required during laparoscopic cholecystectomies in advance, facilitating a smoother surgical workflow and reducing the need for verbal communication. As the instruments in the left working trocar are changed less frequently and according to a standardized procedure, the prediction system works particularly well for this trocar.ConclusionsThe robotic scrub nurse thus acts as a mind reader and helps to mitigate staff shortages by taking over a great share of the workload during surgeries while additionally enabling an enhanced process standardization.

Quality of Experience Oriented Cross-Layer Optimization for Real-Time XR Video Transmission

Quality of Experience Oriented Cross-Layer Optimization for Real-Time XR Video Transmission

Design and Implementation of High-Performance FPGA Accelerator for Non-Separable Discrete Fourier Transform Optimizing Real-Time Image and Video Processing

Design and Implementation of High-Performance FPGA Accelerator for Non-Separable Discrete Fourier Transform Optimizing Real-Time Image and Video Processing

AI-ENHANCED TRACKSEGNET AN ADVANCED MACHINE LEARNING TECHNIQUE FOR VIDEO SEGMENTATION AND OBJECT TRACKING

Video segmentation and object tracking are critical tasks in computer vision with applications spanning surveillance, autonomous driving, and interactive media. Traditional methods often struggle with the dynamic nature of video data, where object occlusions, variations in illumination, and complex motion patterns present significant challenges. Existing segmentation and tracking systems frequently suffer from inaccuracies in handling real-time video sequences, particularly in distinguishing and tracking multiple overlapping objects. The limitations of current models in addressing these issues necessitate the development of more advanced techniques that can effectively manage dynamic scenes and improve tracking accuracy. To address these challenges, we propose an advanced machine learning technique, AI-Enhanced TrackSegNet, which integrates deep learning with novel attention mechanisms for improved video segmentation and object tracking. Our method utilizes a combination of Convolutional Neural Networks (CNNs) for feature extraction and Long Short-Term Memory (LSTM) networks for temporal sequence modeling. We introduce an attention-based mechanism to dynamically focus on relevant features, enhancing the model's ability to handle occlusions and varying object appearances. The model was trained on a diverse dataset of video sequences, incorporating both synthetic and real-world footage. The AI-Enhanced TrackSegNet demonstrated significant improvements in performance compared to existing techniques. Our method achieved an average Intersection over Union (IoU) score of 86.7% for segmentation and a tracking precision rate of 91.3% on the MOT17 benchmark dataset. These results represent a 10.2% improvement in IoU and a 7.5% increase in tracking precision compared to state-of-the-art methods. The model also exhibited enhanced robustness in complex scenes, handling occlusions and motion variations with greater accuracy.

Vivid-persona: customizable persona tool with interactive and immersive experiences

In the design field, persona is widely used to help design teams understand their target users. However, existing persona tools have limitations such as insufficient interactivity, inability to arouse designers’ empathy, and inability to finely segment users. We developed Vivid-Persona, which is a customisable persona tool offering an interactive and immersive experience. Designers can set the demographics and scenarios of the target user to customise a persona. The tool generates a realistic image of the persona using a Diffusion model. Embedded with the ChatGPT and SadTalker models, the persona can engage in real-time video chats with designers. The persona is augmented with facial expressions, based on the analysis of the emotions in utterances. A user study showed that Vivid-Persona effectively solves the problem of user segmentation, significantly enhancing the designer’s perception of persona in terms of authenticity, clarity, empathy and willingness to use, as well as the quality of the output design solutions.

Performance Evaluation of Lightweight Stream Ciphers for Real-Time Video Feed Encryption on ARM Processor

Performance Evaluation of Lightweight Stream Ciphers for Real-Time Video Feed Encryption on ARM Processor

Active Queue Management in L4S with Asynchronous Advantage Actor-Critic: A FreeBSD Networking Stack Perspective

Bufferbloat is one of the leading causes of high data transmission latency and jitter on the Internet, which severely impacts the performance of low-latency interactive applications such as online streaming, cloud-based gaming/applications, Internet of Things (IoT) applications, voice over IP (VoIP), real-time video conferencing, and so forth. There is currently a pressing need for developing Transmission Control Protocol (TCP) congestion control algorithms and bottleneck queue management schemes that can collaboratively control/reduce end-to-end latency, thus ensuring optimal quality of service (QoS) and quality of experience (QoE) for users. This paper introduces a novel solution by experimentally integrate the low latency, low loss, and scalable throughput (L4S) architecture (specified by the IETF in RFC 9330) in FreeBSD framework with the asynchronous advantage actor-critic (A3C) reinforcement learning algorithm. The first phase involves incorporating a modified dual-queue coupled active queue management (AQM) system for L4S into the FreeBSD networking stack, enhancing queue management and mitigating latency and packet loss. The second phase employs A3C to adjust and fine-tune the system performance dynamically. Finally, we evaluate the proposed solution’s effectiveness through comprehensive experiments, comparing it with traditional AQM-based systems. This paper contributes to the advancement of machine learning (ML) for transport protocol research in the field. The experimental implementation and results presented in this paper are made available through our GitHub repositories.

Lane Detection, Segmentation, Pothole Detection and Traffic Sign Recognition for ADAS

Lane detection, a critical aspect of advanced driver assistance systems (ADAS) and autonomous vehicles, is addressed in this work, combining classical computer vision methods with deep learning techniques. The proposed solution, utilizing a UNet-based model trained with TensorFlow and Keras, enhances vehicle perception for intelligent transportation systems. Integrated functionalities for pothole detection and traffic sign detection further contribute to safety and efficiency, enabling vehicles to identify road hazards and comply with regulations. The system encompasses data preprocessing, model training, and real-time video analysis, while a classical lane detection pipeline using OpenCV showcases various stages such as grayscale conversion, Gaussian blur, Canny edge detection, masking, Hough transform, and lane overlay. This comprehensive approach supports road safety, traffic management, and transportation efficiency initiatives, making significant strides in intelligent transportation systems and urban planning.

SMART BABY MONITORING SYSTEM USING YOLO V8 ALGORITHM

The Smart Baby Monitoring System using the YOLO V8 algorithm is designed to enhance infant monitoring by leveraging advanced computer vision techniques. This project utilizes YOLO (You Only Look Once) version 8, a state-of-the-art object detection algorithm, implemented with Python and frameworks like Tensor Flow or PyTorch, to detect and track objects in real-time video feeds. The system incorporates features for facial recognition to identify known caregivers and alert mechanisms for unusual activities or emergencies. The user interface provides real-time alerts, visualizations, and historical data analysis for caregivers via a web or mobile application. By leveraging YOLO V8's efficiency in object detection and Python's capabilities for data processing and integration, this system aims to enhance safety, improve care giving efficiency, and provide peace of mind to parents and caregivers.

Artificial intelligence model for automated surgical instrument detection and counting: an experimental proof-of-concept study

BackgroundRetained surgical items (RSI) are preventable events that pose a significant risk to patient safety. Current strategies for preventing RSIs rely heavily on manual instrument counting methods, which are prone to human error. This study evaluates the feasibility and performance of a deep learning-based computer vision model for automated surgical tool detection and counting.MethodsA novel dataset of 1,004 images containing 13,213 surgical tools across 11 categories was developed. The dataset was split into training, validation, and test sets at a 60:20:20 ratio. An artificial intelligence (AI) model was trained on the dataset, and the model’s performance was evaluated using standard object detection metrics, including precision and recall. To simulate a real-world surgical setting, model performance was also evaluated in a dynamic surgical video of instruments being moved in real-time.ResultsThe model demonstrated high precision (98.5%) and recall (99.9%) in distinguishing surgical tools from the background. It also exhibited excellent performance in differentiating between various surgical tools, with precision ranging from 94.0 to 100% and recall ranging from 97.1 to 100% across 11 tool categories. The model maintained strong performance on a subset of test images containing overlapping tools (precision range: 89.6–100%, and recall range 97.2–98.2%). In a real-time surgical video analysis, the model maintained a correct surgical tool count in all non-transition frames, with a median inference speed of 40.4 frames per second (interquartile range: 4.9).ConclusionThis study demonstrates that using a deep learning-based computer vision model for automated surgical tool detection and counting is feasible. The model’s high precision and real-time inference capabilities highlight its potential to serve as an AI safeguard to potentially improve patient safety and reduce manual burden on surgical staff. Further validation in clinical settings is warranted.

Acquiring basketball plays through varied speeds of video demonstration: Effect of time of day

ABSTRACT The present study aimed to elucidate the effects of real-time versus slow-motion videos, as a function of time of day, on the acquisition of basketball plays. Fifty first-year sports science students (Mage = 18.82 y, SD = 0.49) were quasi-randomly allocated to either into 0.5 or 1.0 times normal video speed, attending morning (08:00–09:00 h) and late-afternoon (16:00–17:00 h) sessions. After measuring oral temperature (OT) and mood states (MS), participants observed a basketball system’s evolution twice and completed two recall tests [game comprehension (GC) and game performance (GP) tests]. Afternoon sessions exacerbated higher OT, higher negative MS (e.g. anxiety and fatigue), and lower positive MS (i.e. vigor) compared to mornings (regardless of video speed). GC and GP were higher in the morning than in the afternoon, with real-time (p < 0.001, both) and slow-motion videos (p < 0.01 and p < 0.001, respectively). GC and GP were higher with slow-motion videos compared to real-time videos in morning and afternoon sessions (p < 0.01, both). The findings highlight the morning’s superiority in the acquisition of motor knowledge from videos, due to mood disturbances and lower arousal levels. Additionally, the findings encourage teachers/coaches to communicate basketball plays through slow-motion videos, particularly in the morning.

Semantic segmentation algorithm for video from UAV based on adaptive keyframe scheduling via similarity measurement

Abstract Unmanned aerial vehicle (UAV) videos exhibit complex object motion features and significant differences between frame features. To solve the problems of feature information loss and drastic accuracy decline in applying the video semantic segmentation method based on the fixed-period update strategy of keyframes, a keyframe recognition method based on similarity measurement is proposed, forming a video semantic segmentation algorithm based on adaptive keyframe scheduling. one keyframe recognition method based on pixel similarity measurement is established by modeling the similarity relationship between low-level pixels in adjacent frames. Meanwhile, the other keyframe recognition method based on feature similarity measurement is established by constructing a shallow Siamese network to measure the similarity relationship between features of frames. Then a discriminative network is constructed based on the obtained similarity of inter frames, and the segmentation process is accelerated by reusing features of keyframes through the optical flow network. Thereby a video semantic segmentation method for UAV based on adaptive keyframe strategy is established. The effectiveness of the proposed video semantic segmentation algorithm is verified on UAVid dataset. The results demonstrated that the speed of the proposed algorithm reaches 53.2 FPS and 54.5 FPS on the premise that the mean intersection over union is higher than 40% (this value is compared with the segmentation accuracy in the balanced mode when the similarity threshold is 0.76 and 0.88, respectively), which is 18.5 FPS and 19.8 FPS higher than the PSPNet image semantic segmentation algorithm. In addition, analysis of experiment results shows that the pixel-similarity-based keyframe recognition is suitable for high-precision video semantic segmentation scenes that need to improve segmentation efficiency, and the feature-similarity-based keyframe recognition is more suitable for high real-time video semantic segmentation tasks that require a small decrease in overall algorithm accuracy. &amp;#xD;In a word, the proposed video semantic segmentation algorithm based on adaptive keyframe scheduling via similarity measurement can improve segmentation speed while ensuring segmentation accuracy and stability.&amp;#xD;

A YOLOv7-Based Method for Ship Detection in Videos of Drones

With the rapid development of the shipping industry, the number of ships is continuously increasing, and maritime accidents happen frequently. In recent years, computer vision and drone flight control technology have continuously developed, making drones widely used in related fields such as maritime target detection. Compared to the cameras fixed on ships, a greater flexibility and a wider field of view is provided by cameras equipped on drones. However, there are still some challenges in high-altitude detection with drones. Firstly, from a top-down view, the shapes of ships are very different from ordinary views. Secondly, it is difficult to achieve faster detection speeds because of limited computing resources. To solve these problems, we propose YOLOv7-DyGSConv, a deep learning-based model for detecting ships in real-time videos captured by drones. The model is built on YOLOv7 with an attention mechanism, which enhances the ability to capture targets. Furthermore, the Conv in the Neck of the YOLOv7 model is replaced with the GSConv, which reduces the complexity of the model and improves the detection speed and detection accuracy. In addition, to compensate for the scarcity of ship datasets in top-down views, a ship detection dataset containing 2842 images taken by drones or with a top-down view is constructed in the research. We conducted experiments on our dataset, and the results showed that the proposed model reduced the parameters by 16.2%, the detection accuracy increased by 3.4%, and the detection speed increased by 13.3% compared with YOLOv7.

Lane Line and Object Detection Using Yolo v3

In the contemporary age, creating autonomous vehi- cles is a crucial starting point for the advancement of intelligent transportation systems that rely on sophisticated telecommu- nications network infrastructure, including the upcoming 6g networks. The paper discusses two significant issues, namely, lane detection and obstacle detection (such as road signs, traffic lights, vehicles ahead, etc.) using image processing algorithms. To address issues like low accuracy in traditional image processing methods and slow real-time performance of deep learning-based methods, barriers for smart traffic lane and object detection algorithms are proposed. We initially rectify the distorted image resulting from the camera and then employ a threshold algorithm for the lane detection algorithm. The image is obtained by extracting a specific region of interest and applying an inverse perspective transform to obtain a top-down view. Finally, we apply the sliding window technique to identify pixels that belong to each lane and modify it to fit a quadratic equation. The Yolo algorithm is well-suited for identifying various types of obstacles, making it a valuable tool for solving identification problems. Finally, we utilize real-time videos and a straightforward dataset to conduct simulations for the proposed algorithm. The simula- tion outcomes indicate that the accuracy of the proposed method for lane detection is 97.91% and the processing time is 0.0021 seconds. The proposal for detecting obstacles has an accuracy rate of 81.90% and takes only 0.022 seconds to process. Compared to the conventional image processing technique, the proposed method achieves an average accuracy of 89.90% and execution time of 0.024 seconds, demonstrating a robust capability against noise. The findings demonstrate that the suggested algorithm can be implemented in self-driving car systems, allowing for efficient and fast processing of the advanced network.

The effects of telepresence with real-time video and audio communication on parent-infant interaction and staff experience in neonatal intensive care unit

ObjectiveWe implemented an online visitation system named “telepresence” in the neonatal intensive care unit (NICU) for family members at home to communicate with their babies in real-time using video and audio. This study evaluated the impact of this system on families and medical staff. MethodsNineteen families of babies admitted to the NICU between 2022 and 2023 and 65 medical staff participated. Each family experienced two weeks of virtual visits. Changes in parental depression and attachment were assessed. ResultBefore and after telepresence, the median Edinburgh Postnatal Depression Scale score reduced from 6 to 4 (p = 0.026), and the Mother-to-Infant Bonding Scale score showed a decreasing trend, with both medians at 2 (p = 0.057). Eighty-nine percent of the parents and 97% of staff reported that telepresence did not increase parental stress, and 88% of parents felt positive changes in their baby's siblings. All parents wanted to visit their babies in person after seeing them on camera. ConclusionTelepresence improved parental mental health, reduced family distress, and supported connection with their infants, making them eager to visit in person. InnovationsThis technology potentially make parents want to visit more by helping them feel more connected to their infants.

DSJA: Distributed Server-Driven Joint Route Scheduling and Streaming Adaptation for Multi-Party Realtime Video Streaming

DSJA: Distributed Server-Driven Joint Route Scheduling and Streaming Adaptation for Multi-Party Realtime Video Streaming