Real-time Video Processing Research Articles

Real-Time Video Processing for Measuring Zigzag Length of Pantograph–Catenary Systems Based on GPS Correlation

Recent years have seen outstanding developments in research and technology, highlighting the importance of railway transportation, especially the implementation of high-speed trains, which is becoming more and more challenging. This facilitates extensive research into the science and technology of the electrical interaction between the components of pantograph–catenary systems (PCSs). Problems regarding the PCS can result in infrastructure incidents, potentially stopping train operations. A common cause of failure in electrified railway PCS is a contact wire’s zigzag length that exceeds the prescribed technical limit, which can be caused by missing droppers or faults in the mounting mechanism. This work proposes a video camera-based monitoring technique for zigzag geometry measurement that additionally employs a Global Positioning System (GPS) sensor to detect the current geographical position of the point of zigzag length measurement. There are two proposed techniques for measuring the length of the zigzag based on image processing. In the first technique, using previously recorded data, the images are analyzed in the laboratory, and in the second, the images are analyzed in real time. Based on the results, we suggest a model and prediction of zigzag length employing hybrid deep neural networks.

Real-time Traffic Conflict Prediction at Signalized Intersections Using Vehicle Trajectory Data and Deep Learning

Real-time conflict prediction at signalized intersections is crucial for urban road safety management. This study developed a real-time conflict prediction framework for signalized intersections using video data real-time recognition technology and deep learning techniques, incorporating lane-level information and feature interactions. The modeling framework consists of three stages: real-time video data extraction and processing, the development of a Deep and Cross Network (DCN)-based real-time traffic conflict prediction model, and conflict-driven factor interpretability analysis through SHapley Additive exPlanations (SHAP). In the first stage, an efficient automated trajectory extraction system is designed to obtain vehicle trajectories in real time for dynamic traffic parameters and conflict frequency identification. In the second stage, a DCN model is developed to construct the relationships between dynamic traffic parameters, including their interactions, and traffic conflicts. In the third stage, SHAP explores the impact mechanisms of different dynamic traffic parameters on traffic conflicts. The model’s predictive performance and interpretability are evaluated using intersection video data from Changsha City, China. The results show that: 1) In real-time traffic conflict prediction at signalized intersections across different modified time-to-conflict thresholds (1.5s and 3.0s), the DCN model consistently outperformed statistical and machine learning models. 2) High traffic flow on main and secondary roads at signalized intersections significantly increases the complexity and frequency of conflicts, with varying sensitivity depending on the interaction of traffic flow, speed, and platoon length. 3) The proposed framework provides a safety measurement standard for data-driven road safety management methods.

Smart structural health monitoring using computer vision and edge computing

Structural health monitoring (SHM) provides real-time data on the condition and performance of infrastructure, enabling timely and cost-effective maintenance interventions, and hence enhanced safety and extended service life. The computer vision-based non-contact sensor has emerged as a promising alternative to conventional contact-type sensors for structural displacement measurement and SHM. Many of the currently reported vision-based structural displacement measurement systems typically temporarily set up a video camera from a distance to the structure. The collected images or videos are usually stored locally and post-processed offline to obtain structural displacement responses, which is cumbersome and limited to short-term SHM applications. The recent development of technologies empowered by the Internet of Things (IoT) and edge computing has enabled real-time video processing and analysis at the source, minimizing latency, reducing bandwidth requirements, and enabling prompt decision-making, thereby enhancing efficiency and responsiveness compared to traditional offline video recording and processing systems. In this paper, an edge computing vision-based displacement measurement system (EdgeCVDMS) is developed. Video recording, processing, and displacement response identification are entirely performed on an edge device integrated with the vision-based displacement tracking algorithm, thereby greatly reducing the amount of data transmitted to the cloud server. The feasibility and applicability of the developed sensing system are experimentally validated on a laboratory-scaled transmission tower structure. The proposed EdgeCVDMS is cost-effective, easily deployable, and of great potential to be applied for the condition assessment of a larger population of aging civil infrastructure.

Design and Implementation of High-Performance FPGA Accelerator for Non-Separable Discrete Fourier Transform Optimizing Real-Time Image and Video Processing

Design and Implementation of High-Performance FPGA Accelerator for Non-Separable Discrete Fourier Transform Optimizing Real-Time Image and Video Processing

A YOLO-based deep learning model for Real-Time face mask detection via drone surveillance in public spaces

Automating face mask detection in public areas is paramount to maintaining public health, especially in the context of the COVID-19 pandemic. Utilization of technologies such as deep learning and computer vision systems enables effective monitoring of mask compliance, thereby minimizing the risk of virus spread. Real-time detection helps in prompt intervention for and enforcement of the use of masks, thereby preventing potential outbreaks and ensuring compliance with public health guidelines. This method helps save human resources and makes the reinforcement of wearing masks in public areas consistent and objective. Automatic detection of face masks serves as a key tool for preventing the spread of contagious diseases, protecting public health, and creating a safer environment for every person. This study addresses the challenges of real-time face mask detection via drone surveillance in public spaces, with reference to three categories: wearing of mask, incorrect wearing of mask, and no mask. Addressing these challenges entails an efficient and robust object detection and recognition algorithm. This algorithm can deal with a crowd of multiple faces via a mobile camera carried by a mini drone, and performs real-time video processing. Accordingly, this study proposes a You Only Look Once (YOLO) based deep learning C-Mask model for real-time face mask detection and recognition via drone surveillance in public spaces. The C-Mask model aims to operate within a mini drone surveillance system and provide efficient and robust face mask detection. The C-Mask model performs preprocessing, feature extraction, feature generation, feature enhancement, feature selection, and multivariate classification tasks for each face mask detection cycle. The preprocessing task prepares the training and testing data in the form of images for further processing. The feature extraction task is performed using a Convolutional Neural Network (CNN). Moreover, Cross-Stage Partial (CSP) DarkNet53 is used to improve the feature extraction and to facilitate the model’s object detection ability. A data augmentation algorithm is used for feature generation to enhance the model’s training robustness. The feature enhancement task is performed by applying the Path Aggregation Network (PANet) and Spatial Pyramid Pooling Network (SPPNet) algorithms, which are deployed to enhance the extracted and generated features. The classification task is performed through multi-label classification, wherein each object in an image can belong to multiple classes simultaneously, and the network generates a grid of bounding boxes and corresponding confidence scores for each class. The YOLO-based C-Mask model testing is performed by experimenting with various face mask detection scenarios and with varying mask colors and types, to ensure the efficiency and robustness of the proposed model. The C-Mask model test results show that this model can correctly and effectively detect face masks in real-time video streams under various conditions with an overall accuracy of 92.20%, precision of 92.04, recall of 90.83%, and F1-score of 89.95%, for all the three classes. These high scores have been obtained despite mini drone mobility and camera orientation adjustment substantially affecting face mask detection performance.

Recognizing online video genres using ensemble deep convolutional learning for digital media service management

It's evident that streaming services increasingly seek to automate the generation of film genres, a factor profoundly shaping a film's structure and target audience. Integrating a hybrid convolutional network into service management emerges as a valuable technique for discerning various video formats. This innovative approach not only categorizes video content but also facilitates personalized recommendations, content filtering, and targeted advertising. Given the tendency of films to blend elements from multiple genres, there is a growing demand for a real-time video classification system integrated with social media networks. Leveraging deep learning, we introduce a novel architecture for identifying and categorizing video film genres. Our approach utilizes an ensemble gated recurrent unit (ensGRU) neural network, effectively analyzing motion, spatial information, and temporal relationships. Additionally,w we present a sophisticated deep neural network incorporating the recommended GRU for video genre classification. The adoption of a dual-model strategy allows the network to capture robust video representations, leading to exceptional performance in multi-class movie classification. Evaluations conducted on well-known datasets, such as the LMTD dataset, consistently demonstrate the high performance of the proposed GRU model. This integrated model effectively extracts and learns features related to motion, spatial location, and temporal dynamics. Furthermore, the effectiveness of the proposed technique is validated using an engine block assembly dataset. Following the implementation of the enhanced architecture, the movie genre categorization system exhibits substantial improvements on the LMTD dataset, outperforming advanced models while requiring less computing power. With an impressive F1 score of 0.9102 and an accuracy rate of 94.4%, the recommended model consistently delivers outstanding results. Comparative evaluations underscore the accuracy and effectiveness of our proposed model in accurately identifying and classifying video genres, effectively extracting contextual information from video descriptors. Additionally, by integrating edge processing capabilities, our system achieves optimal real-time video processing and analysis, further enhancing its performance and relevance in dynamic media environments.

Edge computing based real-time Nephrops (Nephrops norvegicus) catch estimation in demersal trawls using object detection models

In demersal trawl fisheries, the unavailability of the catch information until the end of the catching process is a drawback, leading to seabed impacts, bycatches and reducing the economic performance of the fisheries. The emergence of in-trawl cameras to observe catches in real-time can provide such information. This data needs to be processed in real-time to determine the catch compositions and rates, eventually improving sustainability and economic performance of the fisheries. In this study, a real-time underwater video processing system counting the Nephrops individuals entering the trawl has been developed using object detection and tracking methods on an edge device (NVIDIA Jetson AGX Orin). Seven state-of-the-art YOLO models were tested to discover the appropriate training settings and YOLO model. To achieve real-time processing and accurate counting simultaneously, four frame skipping ideas were evaluated. It has been shown that adaptive frame skipping approach, together with YOLOv8s model, can increase the processing speed up to 97.47 FPS while achieving correct count rate and F-score of 82.57% and 0.86, respectively. In conclusion, this system can improve the sustainability of the Nephrops directed trawl fishery by providing catch information in real-time.

Silicon photonic neuromorphic accelerator using integrated coherent transmit-receive optical sub-assemblies

Neural networks, having achieved breakthroughs in many applications, require extensive convolutions and matrix-vector multiplication operations. To accelerate these operations, benefiting from power efficiency, low latency, large bandwidth, massive parallelism, and CMOS compatibility, silicon photonic neural networks have been proposed as a promising solution. In this study, we propose a scalable architecture based on a silicon photonic integrated circuit and optical frequency combs to offer high computing speed and power efficiency. A proof-of-concept silicon photonics neuromorphic accelerator based on integrated coherent transmit–receive optical sub-assemblies, operating over 1TOPS with only one computing cell, is experimentally demonstrated. We apply it to process fully connected and convolutional neural networks, achieving a competitive inference accuracy of up to 96.67% in handwritten digit recognition compared to its electronic counterpart. By leveraging optical frequency combs, the approach’s computing speed is possibly scalable with the square of the cell number to realize over 1 Peta-Op/s. This scalability opens possibilities for applications such as autonomous vehicles, real-time video processing, and other high-performance computing tasks.

ZYNQ-Based Visible Light Defogging System Design Realization.

Under a foggy environment, the air contains a large number of suspended particles, which lead to the loss of image information and decline of contrast collected by the vision system. This makes subsequent processing and analysis difficult. At the same time, the current stage of the defogging system has problems such as high hardware cost and poor real-time processing. In this article, an image defogging system is designed based on the ZYNQ platform. First of all, on the basis of the traditional dark-channel defogging algorithm, an algorithm for segmenting the sky is proposed, and in this way, the image distortion caused by the sky region is avoided, and the atmospheric light value and transmittance are estimated more accurately. Then color balancing is performed after image defogging to improve the quality of the final output image. The parallel computing advantage and logic resources of the PL (Programmable Logic) part (FPGA) of ZYNQ are fully utilized through instruction constraints and logic optimization. Finally, the visible light detector is used as the input to build a real-time video processing experiment platform. The experimental results show that the system has a good defogging effect and meet the real-time requirements.

Intelligent Methods for Forest Fire Detection Using Unmanned Aerial Vehicles

This research addresses the problem of early detection of smoke and open fire on the observed territory by unmanned aerial vehicles. We solve the tasks of improving the quality of incoming video data by removing motion blur and stabilizing the video stream; detecting the horizon line in the frame; and identifying fires using semantic segmentation with Euclidean–Mahalanobis distance and the modified convolutional neural network YOLO. The proposed horizon line detection algorithm allows for cutting off unnecessary information such as cloud-covered areas in the frame by calculating local contrast, which is equivalent to the pixel informativeness indicator of the image. Proposed preprocessing methods give a delay of no more than 0.03 s due to the use of a pipeline method for data processing. Experimental results show that the horizon clipping algorithm improves fire and smoke detection accuracy by approximately 11%. The best results with the neural network were achieved with YOLO 5m, which yielded an F1 score of 76.75% combined with a processing speed of 45 frames per second. The obtained results differ from existing analogs by utilizing a comprehensive approach to early fire detection, which includes image enhancement and alternative real-time video processing methods.

Towards adversarial robustness verification of no-reference image- and video-quality metrics

In this paper, we propose a new method of analysing the stability of modern deep image- and video-quality metrics to different adversarial attacks. The stability analysis of quality metrics is becoming important because nowadays the majority of metrics employ neural networks. Unlike traditional quality metrics based on nature scene statistics or other hand-crafter features, learning-based methods are more vulnerable to adversarial attacks. The usage of such unstable metrics in benchmarks may lead to being exploited by the developers of image and video processing algorithms to achieve higher positions in leaderboards. The majority of known adversarial attacks on images designed for computer vision tasks are not fast enough to be used within real-time video processing algorithms. We propose four fast attacks on metrics suitable for real-life scenarios. The proposed methods are based on creating perturbations that increase metrics scores and can be applied frame-by-frame to attack videos. We analyse the stability of seven widely used no-reference image- and video-quality metrics to proposed attacks. The results showed that only three metrics are stable against our real-life attacks. This research yields insights to further aid in designing stable neural-network-based no-reference quality metrics. Proposed attacks can serve as an additional verification of metrics’ reliability.

Face Mask Detection System Using Machine Learning Algorithms

The project presented in this report is a real-time face mask detection system using computer vision and deep learning techniques. The primary objective of this project is to develop a system that can detect whether a person is wearing a face mask or not, with a focus on real-time performance.
 The system utilizes pre-trained deep learning models for face detection and mask classification. It leverages the MobileNetV2 architecture as a feature extractor and deploys the model in real-time video streams. When a face is detected, the system classifies it as "Mask" or "No Mask" with associated confidence scores. The project involves key components, including the use of OpenCV for image processing and real-time video capture, TensorFlow/ Keras for deep learning, and the integration of pre-trained models. The code is well- structured, and it demonstrates proficiency in model loading, image preprocessing, and real-time video processing.
 The findings of the project showcase a practical application for face mask detection, which has gained significance in the context of public health and safety. The system provides a valuable tool for monitoring mask compliance in public spaces and can contribute to efforts to mitigate the spread of contagious diseases. The project demonstrates the importance of combining computer vision and deep learning in real-world applications, and it serves as a reference for those interested in similar projects or applications in the field of image processing and object detection. In summary, this project illustrates the successful implementation of a real-time face mask detection system and underscores its potential contributions to public health and safety measures.

Bounded PCA based Multi Sensor Image Fusion Employing Curvelet Transform Coefficients

The fusion of thermal and visible images acts as an important device for target detection. The quality of the spectral content of the fused image improves with wavelet-based image fusion. However, compared to PCA-based fusion, most wavelet-based methods provide results with a lower spatial resolution. The outcome gets better when the two approaches are combined, but they may still be refined. Compared to wavelets, the curvelet transforms more accurately depict the edges in the image. Enhancing the edges is a smart way to improve spatial resolution and the edges are crucial for interpreting the images. The fusion technique that utilizes curvelets enables the provision of additional data in both spectral and spatial areas concurrently. In this paper, we employ an amalgamation of Curvelet Transform and a Bounded PCA (CTBPCA) method to fuse thermal and visible images. To evidence the enhanced efficiency of our proposed technique, multiple evaluation metrics and comparisons with existing image merging methods are employed. Our approach outperforms others in both qualitative and quantitative analysis, except for runtime performance. Future Enhancement-The study will be based on using the fused image for target recognition. Future work should also focus on this method’s continued improvement and optimization for real-time video processing.

A Fast Gradient Iterative Affine Motion Estimation Algorithm Based on Edge Detection for Versatile Video Coding

In the Versatile Video Coding (VVC) standard, affine motion models have been applied to enhance the resolution of complex motion patterns. However, due to the high computational complexity involved in affine motion estimation, real-time video processing applications face significant challenges. This paper focuses on optimizing affine motion estimation algorithms in the VVC environment and proposes a fast gradient iterative algorithm based on edge detection for efficient computation. Firstly, we establish judging conditions during the construction of affine motion candidate lists to streamline the redundant judging process. Secondly, we employ the Canny edge detection method for gradient assessment in the affine motion estimation process, thereby enhancing the iteration speed of affine motion vectors. The experimentalresults show that the encoding time of the affine motion estimation algorithm is about 15–35% lower than the overall encoding time of the anchor algorithm encoder, the average encoding time of the affine motion estimation part of the inter-frame prediction part is reduced by 24.79%, and the peak signal-to-noise ratio (PSNR) is only reduced by 0.04.

Guided Linear Upsampling

Guided upsampling is an effective approach for accelerating high-resolution image processing. In this paper, we propose a simple yet effective guided upsampling method. Each pixel in the high-resolution image is represented as a linear interpolation of two low-resolution pixels, whose indices and weights are optimized to minimize the upsampling error. The downsampling can be jointly optimized in order to prevent missing small isolated regions. Our method can be derived from the color line model and local color transformations. Compared to previous methods, our method can better preserve detail effects while suppressing artifacts such as bleeding and blurring. It is efficient, easy to implement, and free of sensitive parameters. We evaluate the proposed method with a wide range of image operators, and show its advantages through quantitative and qualitative analysis. We demonstrate the advantages of our method for both interactive image editing and real-time high-resolution video processing. In particular, for interactive editing, the joint optimization can be precomputed, thus allowing for instant feedback without hardware acceleration.

A Novel Memory Concurrent Editing Model for Large-Scale Video Streams in Edge Computing

Efficient management and utilization of edge server memory buffers are crucial for improving the efficiency of concurrent editing in the concurrent editing application scenario of large-scale video in edge computing. In order to elevate the efficiency of concurrent editing and the satisfaction of service users under the constraint of limited memory buffer resources, the allocation of memory buffers of concurrent editing servers is transformed into the bin-packing problem, which is solved using an ant colony algorithm to achieve the least loaded utilization batch. Meanwhile, a new distributed online concurrent editing algorithm for video streams is designed for the conflict problem of large-scale video editing in an edge computing environment. It incorporates dual-buffer read-and-write technology to solve the difficult problem of concurrent inefficiency of editing and writing disks. The experimental results of the simulation show that the scheme not only achieves a good performance in the scheduling of concurrent editing but also implements the editing resource allocation function in an efficient and reasonable way. Compared with the benchmark traditional single-exclusive editing scheme, the proposed optimized scheme can simultaneously enhance editing efficiency and user satisfaction under the restriction of providing the same memory buffer computing resources. The proposed model has a wide application to video real-time processing application scenarios in edge computing.

Deep Learning Inferencing with High-performance Hardware Accelerators

As computer architectures continue to integrate application-specific hardware, it is critical to understand the relative performance of devices for maximum app acceleration. The goal of benchmarking suites, such as MLPerf for analyzing machine learning (ML) hardware performance, is to standardize a fair comparison of different hardware architectures. However, there are many apps that are not well represented by these standards that require different workloads, such as ML models and datasets, to achieve similar goals. Additionally, many apps, like real-time video processing, are focused on latency of computations rather than strictly on throughput. This research analyzes multiple compute architectures that feature ML-specific hardware on a case study of handwritten Chinese character recognition. Specifically, AlexNet and a custom version of GoogLeNet are benchmarked in terms of their streaming latency and maximum throughput for optical character recognition. Considering that these models are composed of fundamental neural network operations yet architecturally different from each other, these models can stress devices in different yet insightful ways that generalizations of the performance of other models can be drawn from. Many devices featuring ML-specific hardware and optimizations are analyzed including Intel and AMD CPUs, Xilinx and Intel FPGAs, NVIDIA GPUs, and Google TPUs. Overall, ML-oriented hardware added to the Intel Xeon CPUs helps to boost throughput by 3.7× and to reduce latency by up to 34.7×, which makes the latency of Intel Xeon CPUs competitive on more parallel models. The TPU devices were limited in terms of throughput due to large data transfer times and not competitive in terms of latency. The FPGA frameworks showcase the lowest latency on the Xilinx Alveo U200 FPGA achieving 0.48 ms on AlexNet using Mipsology Zebra and 0.39 ms on GoogLeNet using Vitis-AI. Through their custom acceleration datapaths coupled with high-performance SRAM, the FPGAs are able to keep critical model data closer to processing elements for lower latency. The massively parallel and high-memory GPU devices with Tensor Core accelerators achieve the best throughput. The NVIDIA Tesla A100 GPU showcases the highest throughput at 42,513 and 52,484 images/second for AlexNet and GoogLeNet, respectively. 1

Object Detection Using Adaptive Block Partition and RCNN Algorithm

Advancements in image and video processing are growing over the years for industrial robots, autonomous vehicles, cryptography, surveillance, medical imaging and computer-human interaction applications. One of the major challenges in real-time image and video processing is the execution of complex functions and high computational tasks. To overcome this issue, a hardware acceleration of different filter algorithms for both image and video processing is implemented on Xilinx Zynq®-7000 System on-Chip (SoC) device consists of Dual-core Cortex™-A9 processors which provides computing ability to perform with the help of software libraries using Vivado® High-Level Synthesis (HLS). The acceleration of object detection algorithms include Sobel-Feldman filter, posterize and threshold filter algorithms implemented with 1920 x 1080 image resolutions for real-time object detection. The implementation results exhibit effective resource utilization such as 45.6% of logic cells, 51% of Look-up tables (LUTs), 29.47% of Flipflops, 15% of Block RAMs and 23.63% of DSP slices under 100 MHz frequency on comparing with previous works. There are a few reasons why tracking is preferable over detecting objects in each frame. Tracking facilitates in identifying the identity of various items across frames when there are several objects. Object detection may fail in some instances, but tracking may still be achievable which takes into account the location and appearance of the object in the previous frame. The key hurdles in real-time image and video processing applications are object tracking and motion detection. Some tracking algorithms are extremely fast because they perform a local search rather than a global search. Tracking algorithms such as meanshift, Regional Neural Network probabilistic data association, particle filter, nearest neighbor, Kalman filter and interactive multiple model (IMM) are available to estimate and predict the state of a system.

An investigation of camera movements and capture techniques on optical flow for real-time rendering and presentation

New and interesting uses for portable devices include the creation and viewing of 3D models and 360-degree photos of real landscapes. To provide a 3D model and a 360-degree view of a scenario, these apps search for real-time rendering and presentation. This study examines the impact of real-time image processing on movements in camera view and the application of optical flow algorithms. The optical flows are affected by how the objects involved motion. The way the image is recorded and the camera movements affect the relative motion. As a consequence, camera motions are responsive to optical flow algorithms. To record an image, the camera may pan around, move in a straight path, or move randomly. We have captured datasets of videos produced in different contexts to better replicate real-world scenarios. Each dataset was captured in a variety of illumination situations, camera movements, and indoor and outdoor recording sites. Here, to determine the most effective optical flow algorithms for use in near real-time applications, such as augmented reality and virtual reality, we compare results based on quality and processing delay for each video frame. We conducted a comparison study to better understand how random camera motion affects real-time video processing. These methods can be used to handle a variety of real-world issues, such as object tracking, video segmentation, structure from motion, gesture tracking, and so on.

HFR-Video-Based Fingertip Velocimeter for Multifinger Tapping Detection

In this study, we propose a novel concept of a software-based fingertip velocimeter using high-frame-rate (HFR) video processing that can simultaneously estimate when and where an operator taps with his/her finger by detecting the high-frequency component that develops when the fingertip actively contacts something. Our software-based fingertip velocimeter can precisely estimate the velocities of multiple fingers through HFR video processing in real time. Digital image correlation (DIC) operating at every frame for sub-pixel-precision velocity estimation is hybridized with CNN-based object detection operating at intervals of dozens of frames to robustly update the fingertip ROI regions during the frame-by-frame DIC operation. We developed a real-time multi-finger tapping detection system that can execute DIC operation on 720×540 resolution images at 500 fps with CNN-based fingertip detection at 30 fps. By presenting several experimental results for finger tapping detection, including virtual keyboard interaction with a ten-finger keyboard input, the effectiveness of our fingertip velocimeter as a finger tapping interface was demonstrated, which can simultaneously estimate the tapping positions and moments of multiple fingers when finger tapping is performed 10 times or more in a second.