Real-time Video Streaming Research Articles

Face Mask Detection System Using Machine Learning Algorithms

The project presented in this report is a real-time face mask detection system using computer vision and deep learning techniques. The primary objective of this project is to develop a system that can detect whether a person is wearing a face mask or not, with a focus on real-time performance.
 The system utilizes pre-trained deep learning models for face detection and mask classification. It leverages the MobileNetV2 architecture as a feature extractor and deploys the model in real-time video streams. When a face is detected, the system classifies it as "Mask" or "No Mask" with associated confidence scores. The project involves key components, including the use of OpenCV for image processing and real-time video capture, TensorFlow/ Keras for deep learning, and the integration of pre-trained models. The code is well- structured, and it demonstrates proficiency in model loading, image preprocessing, and real-time video processing.
 The findings of the project showcase a practical application for face mask detection, which has gained significance in the context of public health and safety. The system provides a valuable tool for monitoring mask compliance in public spaces and can contribute to efforts to mitigate the spread of contagious diseases. The project demonstrates the importance of combining computer vision and deep learning in real-world applications, and it serves as a reference for those interested in similar projects or applications in the field of image processing and object detection. In summary, this project illustrates the successful implementation of a real-time face mask detection system and underscores its potential contributions to public health and safety measures.

“I Can See Your Password”: A Case Study About Cybersecurity Risks in Mid-Air Interactions of Mixed Reality-Based Smart Manufacturing Applications

Abstract This paper aims to present a potential cybersecurity risk existing in mixed reality (MR)-based smart manufacturing applications that decipher digital passwords through a single RGB camera to capture the user’s mid-air gestures. We first created a test bed, which is an MR-based smart factory management system consisting of mid-air gesture-based user interfaces (UIs) on a video see-through MR head-mounted display. To interact with UIs and input information, the user’s hand movements and gestures are tracked by the MR system. We setup the experiment to be the estimation of the password input by users through mid-air hand gestures on a virtual numeric keypad. To achieve this goal, we developed a lightweight machine learning-based hand position tracking and gesture recognition method. This method takes either video streaming or recorded video clips (taken by a single RGB camera in front of the user) as input, where the videos record the users’ hand movements and gestures but not the virtual UIs. With the assumption of the known size, position, and layout of the keypad, the machine learning method estimates the password through hand gesture recognition and finger position detection. The evaluation result indicates the effectiveness of the proposed method, with a high accuracy of 97.03%, 94.06%, and 83.83% for 2-digit, 4-digit, and 6-digit passwords, respectively, using real-time video streaming as input with known length condition. Under the unknown length condition, the proposed method reaches 85.50%, 76.15%, and 77.89% accuracy for 2-digit, 4-digit, and 6-digit passwords, respectively.

Multipurpose deep learning-powered UAV for forest fire prevention and emergency response

This paper presents a customized UAV designed for rescue and safety purposes in the forest sector. The UAV features a durable F450 frame quadcopter with four 1000KV brushless motors and a KK2.1 Flight Control Board for stability and manoeuvrability with a runtime of 90 min. It incorporates a Raspberry Pi camera for real-time video streaming, enabling efficient identification of individuals in need of assistance. The GSM module allows contactless communication, ensuring streamlined and safe interaction. A motor controls the lid of the customizable first aid kit box, facilitating efficient aid delivery. The Neo-6 M GPS module provides accurate localization of the drone and individuals in distress with a horizontal position accuracy of 2.5 m. The UAV collects temperature and humidity data using the DHT 11 sensor having +/- 2 degreesC and +- 5% accuracy respectively. This sensor employs advanced deep learning models, including artificial neural networks (ANN) and generative adversarial networks (GANs), for real-time forest fire prediction with an accuracy of 90.7 % The integration of GANs enhances accuracy through synthetic data generation. Moreover, all these components are interfaced using a Raspberry Pi4 and a GUI, providing a smooth user control experience and end-to-end information for quick and effective emergencyresponse.

ABRF: Adaptive BitRate-FEC Joint Control for Real-Time Video Streaming

Adaptive Forward Error Correction (AFEC) algorithms are proposed to achieve efficient Forward Error Correction (FEC) in Real-Time Communication (RTC). However, current AFEC approaches suffer two key limitations. (1) They do not consider the squeezing effect of redundancy on source bitrate while the squeezing usually happens in real RTC applications with limited bandwidth. (2) They estimate the future packet loss simply, ignoring some critical features of packet loss like randomness and multi-pattern. These drawbacks stop them from providing better Quality of Experience (QoE) in RTC services. We propose ABRF, a general QoE-oriented Adaptive BitRate-FEC joint control algorithm. ABRF makes predictions on the network loss pattern in the coming time and jointly calculates the optimal bitrate-FEC decision based on a QoE model for real-time video streaming. Moreover, ABRF is equipped with a fast adaptation method which helps it generalize across diverse network environments. In terms of Video Multi-method Assessment Fusion (VMAF), experimental results tell that ABRF decreases VMAF degradation caused by packet loss by 68%-95% compared with other AFEC algorithms in real-time video streaming in real-world Internet.

Blockchain Empowered Secure Video Sharing With Access Control for Vehicular Edge Computing

The dramatically growing trend of vehicles equipped with driving camera recorders has allowed realizing real-time crowdsourced video sharing in vehicular edge computing (VEC). Such cameras can assist in monitoring objects directly in front of and behind the vehicles, enabling them to provide important visual information through real-time video streaming in case of possible accidents. Exploiting the on-board units (OBUs) for VEC can allow drivers and passengers to share and access on-road video surveillance services. However, data security and privacy concerns of video generators (owners) are two key challenges that can severely limit video sharing in a VEC environment. In this article, we propose a blockchain empowered publish/subscribe (P/S) scheme to enable one-to-many secure video sharing in the VEC scenario. Then, we design an attribute-based encryption algorithm with static and dynamic attributes (ABE-SD) to achieve fine-grained access control in a mobile environment. Finally, We utilize permissioned blockchain and smart contracts to record access policy and publish and subscribe events, thus resulting in user self-certification and event traceability. The numerical results indicate that our proposed scheme ABE-SD outperforms traditional centralized CP-ABE methods in terms of encryption and decryption performance. The simulation experiments demonstrated that the proposed video-sharing scheme is secure and efficient.

An Analysis of Instagram Live Performance for Health Promotion Strategy: a Case Study of @iakmitangsel

There is a growing interest in leveraging social media platforms for health education and health promotion. This study focuses specifically on Instagram Live, a feature that allows real-time video streaming, to assess its effectiveness in disseminating health-related information and engaging the audience. This research employs a quantitative analysis of engagement metrics of @iakmitangsel’s Instagram Live performance from March to June 2023. The results revealed that Content J had the highest number of views (323) and likes (53). Content I had the lowest number of views (132), while Content B and H had the lowest number of likes (13 each). Content G had the highest number of accounts reached by followers (224), content J had the highest number of accounts reached by non-followers (241), while Content D received the highest number of impressions (561). On the other hand, Content H had the lowest number of accounts reached by follower (118) and Content B had the lowest number of accounts reached by non-follower (27) and the lowest number of impressions (239). The findings of this research will contribute to the knowledge base surrounding the use of Instagram Live as a health promotion strategy, specifically within the context of South Tangerang City. 
 Keywords: social media, engagement metric, health promotion

Automatic real-time fire distance, size and power measurement driven by stereo camera and deep learning

Automatic real-time fire characterization is a crucial requirement of future smart firefighting. This work proposes a novel computer vision method to automatically measure the fire heat release rate, even when the camera is moving in real-time. Firstly, a portable binocular stereo camera is used to capture the real-time fire video stream that is fed into a pre-trained computer-vision model frame-by-frame to detect the fire region. By identifying the fire location inside the image, the real-time distance between the camera and the fire source is determined. This fire distance helps re-scale the images to match the input scale of the AI-image Fire Calorimetry. Then, the deep learning model can automatically output the transient fire power in real time. Results show that the stereo vision system is capable of accurately measuring the distance between the camera and the fire source, flame height, and power, with a relative error of less than 20%. This work provides an automatic and flexible way to measure the distance, power and hazard of fire in real-time, and such a method has broad applications in firefighting operations and decision-making.

DETERMINATION AND COMPARISON METHODS OF BODY POSITIONS ON STREAM VIDEO

Context. One of the tasks of computer vision is the task of determining the human body in the image. There are many methods to solve this problem, some are based on specific equipment (motion capture, kinect) and provide the highest accuracy, some give less accuracy but do not require additional equipment and use less computing power. But usually, such equipment has a high cost, so to ensure the low cost of developments designed to determine the body in the image, you should develop algorithms based on computer vision technology. These algorithms can then be applied to various fields to analyze and compare body positions for a variety of purposes.
 Objective. The aim of the work is to study the effectiveness of existing libraries to determine the human body position in the image, as well as methods for comparing the obtained poses in terms of speed and accuracy of determination.
 Methods. A set of libraries and pose comparison algorithms were analyzed for the purpose of developing a system for determining the correctness of exercise by the user in real time. OpenPose, PoseNet and BlazePose libraries were analyzed for their suitability in recognizing and tracking body parts and movements in real-time video streams. The advantages and disadvantages of each library were evaluated based on their performance, accuracy, and computational efficiency. Additionally, different pose comparison algorithms were analyzed. The effectiveness of each algorithm was evaluated based on their ability to accurately determine and compare body positions.
 As a result, the combination of BlazePose and weighted distance method can achieve the best performance in pose recognition, with high accuracy and robustness across a range of challenging scenarios. The weighted distance method can be further enhanced with techniques such as L2 normalization and pose alignment to improve its accuracy and generalization. Overall, the combination of the BlazePose library and weighted distance methods offers a powerful and effective solution for pose recognition, with high F1 index.
 Results. Existing models for determining poses have shown similar results in the quality of determination with a run-up of about 2%. When developing a cross-platform software product, the BlazePose library, which has an API for working directly in the browser and on mobile platforms, has a significant advantage in speed and accuracy. Also, as the library uses extended 33 keypoint topology it becomes applicable to a wider list of tasks. In the study of comparison methods, the greatest influence on the results was exerted by the quality of pose determination.
 Conclusions. Among the methods of comparison, the method of weighted distances showed the best results. The speed of position determination is inversely proportional to the quality of determination and significantly exceeds the recommended value – 40ms.

Applied Design and Methodology of Delivery Robots Based on Human–Robot Interaction in Smart Cities

This paper proposes an optimised design of an autonomous delivery robot while adopting the latest technologies from the different branching fields of robotics, artificial intelligence, and tele-communication. As a prospective representation of a user-centric robot design, the proposal is design with the major focus on maximizing users’ satisfaction throughout every human–robot interaction (HRI) touchpoints. By the use of sensor fusion techniques along with the deployment of an image-detection-based technique accompanying the point-cloud-detection-based path-planning methodology, the robot delivery would be optimised with effective path-planning and obstacle avoidance capability. With the extension of 5G connectivity, it is proposed that the real-time status update and video stream would enable greater efficiency in terms of remote monitoring and centralised robot administration.

PASS: Patch Automatic Skip Scheme for Efficient Real-Time Video Perception on Edge Devices

Real-time video perception tasks are often challenging over the resource-constrained edge devices due to the concerns of accuracy drop and hardware overhead, where saving computations is the key to performance improvement. Existing methods either rely on domain-specific neural chips or priorly searched models, which require specialized optimization according to different task properties. In this work, we propose a general and task-independent Patch Automatic Skip Scheme (PASS), a novel end-to-end learning pipeline to support diverse video perception settings by decoupling acceleration and tasks. The gist is to capture the temporal similarity across video frames and skip the redundant computations at patch level, where the patch is a non-overlapping square block in visual. PASS equips each convolution layer with a learnable gate to selectively determine which patches could be safely skipped without degrading model accuracy. As to each layer, a desired gate needs to make flexible skip decisions based on intermediate features without any annotations, which cannot be achieved by conventional supervised learning paradigm. To address this challenge, we are the first to construct a tough self-supervisory procedure for optimizing these gates, which learns to extract contrastive representation, i.e., distinguishing similarity and difference, from frame sequence. These high-capacity gates can serve as a plug-and-play module for convolutional neural network (CNN) backbones to implement patch-skippable architectures, and automatically generate proper skip strategy to accelerate different video-based downstream tasks, e.g., outperforming the state-of-the-art MobileHumanPose (MHP) in 3D pose estimation and FairMOT in multiple object tracking, by up to 9.43 times and 12.19 times speedups, respectively. By directly processing the raw data of frames, PASS can generalize to real-time video streams on commodity edge devices, e.g., NVIDIA Jetson Nano, with efficient performance in realistic deployment.

High speed human action recognition using a photonic reservoir computer

The recognition of human actions in videos is one of the most active research fields in computer vision. The canonical approach consists in a more or less complex preprocessing stages of the raw video data, followed by a relatively simple classification algorithm. Here we address recognition of human actions using the reservoir computing algorithm, which allows us to focus on the classifier stage. We introduce a new training method for the reservoir computer, based on “Timesteps Of Interest”, which combines in a simple way short and long time scales. We study the performance of this algorithm using both numerical simulations and a photonic implementation based on a single non-linear node and a delay line on the well known KTH dataset. We solve the task with high accuracy and speed, to the point of allowing for processing multiple video streams in real time. The present work is thus an important step towards developing efficient dedicated hardware for video processing.

JIRA: Joint Incentive Design and Resource Allocation for Edge-Based Real-Time Video Streaming Systems

Edge computing has been introduced as a promising technology for real-time video streaming systems. However, due to the lack of automatic incentives and the limitation of resources, traditional edge computing performs poorly in nowadays scenarios. To handle these two challenges, we propose a framework of Joint Incentive design and Resource Allocation (JIRA) for edge-based real-time video streaming systems. Technically, to ensure the trust and automatic distribution of incentives, we develop a novel smart contract based incentive mechanism and implement a prototype. Meanwhile, we propose an efficient online algorithm, i.e., JIRA, which dynamically adjusts compression ratio, offloading decision, and resource allocation to achieve performance optimization for video streaming under long-term latency and resource constraints. Specifically, JIRA is based on Lyapunov optimization, which decomposes the challenging long-term decision problem into a series of real-time optimization problems. Then we propose a multi-cut <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Generalized Benders Decomposition</i> based algorithm (MGA) to tackle the non-convexity of the decomposed problem. Through rigorous theoretical analysis, we prove the performance bound of JIRA. Extensive simulations demonstrate that the proposed schemes can achieve an efficient trade-off between accuracy performance and energy consumption.

Staying Apart Together from Social

Abstract: The paper presents a methodology for social distancing detection using deep learning to evaluate the distance between people to mitigate the impact of this coronavirus pandemic To Know Social Distancing is the best possible way to detain the spread of Covid Even though the vaccine has been found and working effectively in saving the lives of people, we use Deep Learning with python to monitor social distancing in public places. This is a software tool that monitors if people are maintaining proper social distancing norms or not by analyzing real-time video streams from a CC camera. We use YOLO Model which is trained by the COCO dataset. The detection tool was developed to alert people to maintain a safe distance from each other by evaluating a video feed. The video frame from the camera was used as input and the open-source object detection

Low calculation cost of HEVC coding unit size based on spatial homogeneity detection

For the same video quality, HEVC gives 25% to 50% bitrate savings, compared to its predecessor the Advanced Video Coding H.264 and thus supports resolutions up to 8 K UHD. However, the reduction in bitrates provided by the HEVC leads to an increase in the computational cost of the encoding operation. This complexity can become a true handicap especially for real-time video streaming and also for VANET (Vehicular Ad-Hoc Network) applications such as traffic safety and Video surveillance. The improvement in the bitrates and also the increase in the calculation cost are due to the use of large and multi-sized coding, prediction and transform blocks. Indeed, the H264 coder is based on structure macroblocks with sizes 4 × 4, 8 × 8 and 16 × 16, while H.265 depends on Coding Tree Units (CTUs), CTUs select sizes 4 × 4, 8 × 8, 16 × 16, 32 × 32 and 64 × 64 blocks. This paper proposes a fast CU (Coding Unit) size decision method to reduce the HEVC calculation cost based on spatial homogeneity. Compared with the HM16.13 benchmark test model, the average coding time is reduced by around 40% for CIF / QCIF video sequences, 35% to 43% for class A, B and C test sequences. These important reductions in coding time are obtained with negligible loss of quality and an average increase in bitrates which does not exceed 0.89% for the three configuration modes (All intra, Random Access and Low Delay).

Video2Flink: real-time video partitioning in Apache Flink and the cloud

Video2Flink is a distributed highly scalable video processing system for bounded (i.e., stored) or unbounded (i.e., continuous) and real-time video streams with the same efficiency. It shows how complicated video processing tasks can be expressed and executed as pipelined data flows on Apache Flink, an open-source stream processing platform. Video2Flink uses Apache Kafka to facilitate the machine-to-machine (m2m) communication between the video production and the video processing system that runs on Apache Flink. Features that make the combination of Apache Kafka and Apache Flink a desirable solution to the problem of video processing are the ease of customization, portability, scalability, and fault tolerance. The application is deployed on a Flink cluster of worker machines that run on Kubernetes in the Google Cloud Platform. The experimental results support our claims of speed showing excellent speed-up results for all tested video resolutions. The highest (i.e., more than seven times) speed-up was observed with the videos of the highest resolutions and in real time.

METHOD OF REAL-TIME VIDEO STREAM SYNCHRONIZATION IN THE WORKING ENVIRONMENT OF AN APPLE ORCHARD

Monitoring and analyzing the state of harvest in an apple orchard is essential for efficient horticulture. Unmanned aerial vehicles (UAVs) have been increasingly used for this purpose due to their ability to capture high-resolution images and videos of the orchard from different perspectives. However, synchronizing the video streams from multiple UAVs in real-time presents a significant challenge. The traditional controller-worker architecture used for video stream synchronization is prone to latency issues, which can negatively impact the accuracy of the monitoring system. To address this issue, the authors propose a decentralized method using a consensus algorithm that allows the group of UAVs to synchronize their video streams in real time without relying on a centralized controller device. The proposed method also addresses the challenges of limited network connectivity and environmental factors, such as wind and sunlight. The automated system that utilizes the proposed method was tested in an actual apple orchard. The experimental results show that the proposed approach achieves real-time video stream synchronization with minimal latency and high accuracy. As such, the SSIM index varies from 0.79 to 0.92, with an average value of 0.87, and the PSNR index – varies from 22 to 39, which indicates the decent quality of the received information from combined images. Meanwhile, the effectiveness of the developed system with the proposed approach was proven, which is confirmed by a high average value of 82.69% of the reliability indicator of detecting and calculating the number of fruit fruits and a low average level of type I (14.67%) and II (18.33%) errors. Overall, the proposed method provides a more reliable and efficient approach to real-time video stream synchronization in an apple orchard, which can significantly improve the monitoring and management of apple orchards.

DESIGN AND DEVELOPMENT OF SURVEILLANCE ROVER FOR BORDER SECURITY

Surveillance plays an important role in border areas to keep eye on enemies. In such situations, it is difficult to allow duty of surveillance to a soldier, which may cause danger to the life of one. Rather we can use a surveillance rover to keep eye on border areas. So in such cases this kind of rover is very useful they are small in size and provided with many abilities so they can perform the duty of surveillance and spying perfectly. In case if they found by the combatant, they have no identity to whom they belong. Military on border area are facing many problems so this kind of technology help them to aware about the opponent activities, so they can take further decisions The rovers capabilities include real-time video streaming, image capture allowing operators to quickly assess and respond to changing conditions. With its robust design and maneuverability, the surveillance rover provides a flexible and cost-effective solution for a wide range of surveillance applications, improving safety, and reducing the risks associated with manual inspection and monitoring. KEYWORDS: Surveillance Rover, Image processing, Monitoring, GSM module.

Enhancing Perceptual Experience of Video Quality in Drone Communications by Using VPN Bonding

Recently, drones are increasingly being used as innovative IoT sensors. The application contexts of UAV systems where there is often a need to transmit time-critical data between the drone and the Internet are increasingly numerous and varied. Keeping this in mind and assuming that most professional UAVs are equipped with multiinterface routers and use virtual private networks (VPNs) to ensure reliable, robust, and secure transmission to the operation room, this letter presents a technique to enhance real-time and time-critical communications such as real-time video streaming. An embedded system, Raspberry-based gateway prototype, was used to create a testbed using a professional drone that captured and transmitted real-time video during a 30-min flight over a rural area characterized by poor mobile radio coverage. A perceptual video quality assessment tool, video multimethod assessment fusion (VMAF), has been employed to evaluate the improvement in terms of video quality perceived by the operator.

Application of extended reality technology for real-time navigation in clinical operation

To explore the application of extended reality (XR) technology in clinical surgeries for improving the success rate of surgeries. To assist the surgeons to better understand the location, size and geometric shape of the lesions and reduce potential radiation exposure in minimally invasive surgical navigation based on two-dimensional images, we constructed three-dimensional models based on CT data and used XR technology to achieve intraoperative navigation. An improved quaternion method was used to improve the accuracy of electromagnetic positioning, with which the system error of positioning accuracy was reduced to below 2 mm. A 5G network was used to optimize the server GPU programming algorithm, and real-time video stream coding strategy and network design were adopted to reduce data transmission jam and delay in the remote surgery network, which achieved an average delay of less than 60 ms. A Gaussian distribution deformation model was used to simulate collision detection and stress deformation of the tissues to achieve a tactile perception effect. The intraoperative navigation system based on XR technology allowed more accurate determination of the location of the lesions, effectively reduced the surgical risk, and avoided the risk of intraoperative radiation exposure. The low latency and high fidelity of 5G network achieved real-time interaction during the surgery to provide a technical basis for multi-terminal remote cooperative surgery. The combination of force feedback technology and XR technology enables the surgeons to conduct deep immersion preoperative planning and virtual surgery to improve the success rate of surgery and shorten the learning curve.

Performability Evaluation and Sensitivity Analysis of a Video Streaming on Demand Architecture

In urban air mobility (UAM), video streaming platforms have gained significant attention from media companies due to their growing necessity for on-demand video streaming services-as-you-go in flights. Video streaming services can provide constant data transactions in a huge amount, especially in its operational digital twin (ODT). As a result, the ability to provide a satisfactory user experience through video streaming platforms is critical and complex. This requires continuously operating services while handling numerous user requests for near real-time video streaming. At the same time, high-quality video with high resolution and minimal interruptions is often expected. Therefore, the availability and performance (i.e., performability) of the Back-End video streaming infrastructure are crucial parameters for these platforms. However, evaluating novel video-on-demand architectures in real-world scenarios can be costly due to the numerous parameters involved. Analytical models, such as stochastic Petri nets (SPNs), can serve as an alternative in this complex scenario as they can be used to analyze systems during the design process. In this study, we developed a set of SPN models to assess the performance of a video-on-demand system. These models were designed to illustrate and to evaluate a video-on-demand architecture while considering performance. We had a base SPN model as well as three enhanced variants available. The extended models were generated using the Design of Experiments (DoE) technique and sensitivity analysis results. The DoE identified the factors with the greatest impacts on performance, and the most significant factor interactions. Redundancy strategies were applied to the extended models to increase the availability of the most important components. This redundancy increased the availability of “9 s” from three to five. It is worth noting that this study can help the designers of video streaming systems, to plan and to optimize their ideas based on the provided models.