Video Processing Algorithms Research Articles

Remote photoplethysmography (rPPG) in the wild: Remote heart rate imaging via online webcams

Remote photoplethysmography (rPPG) is a low-cost technique to measure physiological parameters such as heart rate by analyzing videos of a person. There has been growing attention to this technique due to the increased possibilities and demand for running psychological experiments on online platforms. Technological advancements in commercially available cameras and video processing algorithms have led to significant progress in this field. However, despite these advancements, past research indicates that suboptimal video recording conditions can severely compromise the accuracy of rPPG. In this study, we aimed to develop an open-source rPPG methodology and test its performance on videos collected via an online platform, without control of the hardware of the participants and the contextual variables, such as illumination, distance, and motion. Across two experiments, we compared the results of the rPPG extraction methodology to a validated dataset used for rPPG testing. Furthermore, we then collected 231 online video recordings and compared the results of the rPPG extraction to finger pulse oximeter data acquired with a validated mobile heart rate application. Results indicated that the rPPG algorithm was highly accurate, showing a significant degree of convergence with both datasets thus providing an improved tool for recording and analyzing heart rate in online experiments.

Digital Image and Video Processing: Algorithms and Applications

Many of the techniques that are used in digital image and video processing were developed in the 1960s at Bell Laboratories. These techniques have applications in a variety of fields, including medical imaging, videophone, character recognition, satellite imagery, and wire-photo standards conversion. Additional applications include enhancement of photographs or vidoes. The early stages of image and video processing were developed with the intention of enhancing the overall quality of the image or video. For the purpose of enhancing the visual effect of humans, it is intended for human beings. When it comes to image and video processing, the input is an image of poor quality, and the output is an image and video of higher quality. Research on algorithms and applications of digital image and video processing is the primary purpose of this study, which aims to investigate these topics extensively. The methodology employed in this study is qualitative research technique. In accordance with the findings of this research, "Image Processing" refers to the process of analyzing images with the objective of determining the significance of objects and identifying them. Image analysts analyze data that has been remotely sensed and attempt to detect, identify, classify, measure, and evaluate the significance of physical and cultural objects, as well as their patterns and spatial relationships. One subcategory of signal processing is known as video processing, and it is distinguished by the fact that the signals that are input and output are video files or video streams. Technology such as television sets, videocassette recorders (VCRs), DVD players, and other devices all make use of video processing algorithms. The processing of images and videos is extremely useful in a variety of contexts.

Tracking illegal activities using video surveillance systems: a review of the current state of research

The current state of research on the use of the neural networks under martial law to identify offenders committing illegal acts, prevent acts of terrorism, combat sabotage groups in cities, track weapons and control traffic is considered. The methods of detecting illegal actions, weapons, face recognition and traffic violations using video surveillance cameras are analysed. It is proposed to introduce the studied methods into the work of “smart” video surveillance systems in Ukrainian settlements.
 The most effective means of reducing the number of offences is the inevitability of legal liability for offences, so many efforts in law enforcement are aimed at preventing offences. Along with public order policing by patrol police, video surveillance is an effective way to prevent illegal activities in society. Increasing the coverage area of cameras and their number helps to ensure public safety in the area where they are used. However, an increase in the number of cameras creates another problem which is the large amount of video data that needs to be processed. To solve the problem of video data processing, various methods are used, the most modern of which is the use of artificial intelligence to filter a large amount of data from video cameras and the application of various video processing algorithms. The ability to simultaneously process video data from many CCTV cameras without human intervention not only contributes to public safety, but also improves the work of patrol police. The introduction of smart video surveillance systems allows monitoring the situation in public places around the clock, even if there is no police presence in the area.
 In the reviewed studies of video surveillance systems, neural networks, in particular MobileNet V2, YOLO, mYOLOv4-tiny, are used to track illegal actions, criminals and weapons, which are trained on large amounts of video and photo data. It has been found that although neural networks used to require a lot of computing power, they can now be used in IoT systems and smartphones, and this contributes to the fact that more video surveillance devices can be used to monitor the situation.

Towards adversarial robustness verification of no-reference image- and video-quality metrics

In this paper, we propose a new method of analysing the stability of modern deep image- and video-quality metrics to different adversarial attacks. The stability analysis of quality metrics is becoming important because nowadays the majority of metrics employ neural networks. Unlike traditional quality metrics based on nature scene statistics or other hand-crafter features, learning-based methods are more vulnerable to adversarial attacks. The usage of such unstable metrics in benchmarks may lead to being exploited by the developers of image and video processing algorithms to achieve higher positions in leaderboards. The majority of known adversarial attacks on images designed for computer vision tasks are not fast enough to be used within real-time video processing algorithms. We propose four fast attacks on metrics suitable for real-life scenarios. The proposed methods are based on creating perturbations that increase metrics scores and can be applied frame-by-frame to attack videos. We analyse the stability of seven widely used no-reference image- and video-quality metrics to proposed attacks. The results showed that only three metrics are stable against our real-life attacks. This research yields insights to further aid in designing stable neural-network-based no-reference quality metrics. Proposed attacks can serve as an additional verification of metrics’ reliability.

Correlating electrochemical stimulus to structural change in liquid electron microscopy videos using the structural dissimilarity metric

In-situ liquid cell transmission electron microscopy (LCTEM) with electrical biasing capabilities has emerged as an invaluable tool for directly imaging electrode processes with high temporal and spatial resolution. However, accurately quantifying structural changes that occur on the electrode and subsequently correlating them to the applied stimulus remains challenging. Here, we present structural dissimilarity (DSSIM) analysis as segmentation-free video processing algorithm for locally detecting and quantifying structural change occurring in LCTEM videos. In this study, DSSIM analysis is applied to two in-situ LCTEM videos to demonstrate how to implement this algorithm and interpret the results. We show DSSIM analysis can be used as a visualization tool for qualitative data analysis by highlighting structural changes which are easily missed when viewing the raw data. Furthermore, we demonstrate how DSSIM analysis can serve as a quantitative metric and efficiently convert 3-dimensional microscopy videos to 1-dimenional plots which makes it easy to interpret and compare events occurring at different timepoints in a video. In the analyses presented here, DSSIM is used to directly correlate the magnitude and temporal scale of structural change to the features of the applied electrical bias. ImageJ, Python, and MATLAB programs, including a user-friendly interface and accompanying documentation, are published alongside this manuscript to make DSSIM analysis easily accessible to the scientific community.

Detecting Photoepileptic Content: Enhancing Safety

Abstract: Photosensitive epilepsy is a recognized medical condition that affects a sizeable portion of the population, putting individuals at risk for having convulsions triggered by flashing lights or contrasting patterns. Due to its impact on their ability to carry out daily activities and function normally, those who suffer from this condition face a significant challenge. Despite technological advancements and the expansion of the entertainment industry, the prevalence of flashing lights and symmetrical patterns persists, making it imperative to address this issue. This paper seeks to shed light on photosensitive epilepsy and explores a possible solution for detecting photoepileptic content in videos using video processing algorithms with predefined parameters aligned with recommended standards for tolerance levels in photosensitive epilepsy patients. The algorithm takes into account factors such as the luminance, brightness, and duration of flashes deemed safe for people with photosensitive epilepsy. There are currently no automated tools for detecting photoepileptic content in online videos. Some creators manually provide warnings regarding the presence of such content, highlighting the need for a detection tool to protect viewers with photosensitive epilepsy. This paper addresses the lack of regulatory software required to effectively monitor the entertainment industry's rapid evolution. This research contributes to mitigating the effects of photosensitive epilepsy and promoting a safer environment for individuals with this condition by increasing awareness.

Object Detection Using Adaptive Block Partition and RCNN Algorithm

Advancements in image and video processing are growing over the years for industrial robots, autonomous vehicles, cryptography, surveillance, medical imaging and computer-human interaction applications. One of the major challenges in real-time image and video processing is the execution of complex functions and high computational tasks. To overcome this issue, a hardware acceleration of different filter algorithms for both image and video processing is implemented on Xilinx Zynq®-7000 System on-Chip (SoC) device consists of Dual-core Cortex™-A9 processors which provides computing ability to perform with the help of software libraries using Vivado® High-Level Synthesis (HLS). The acceleration of object detection algorithms include Sobel-Feldman filter, posterize and threshold filter algorithms implemented with 1920 x 1080 image resolutions for real-time object detection. The implementation results exhibit effective resource utilization such as 45.6% of logic cells, 51% of Look-up tables (LUTs), 29.47% of Flipflops, 15% of Block RAMs and 23.63% of DSP slices under 100 MHz frequency on comparing with previous works. There are a few reasons why tracking is preferable over detecting objects in each frame. Tracking facilitates in identifying the identity of various items across frames when there are several objects. Object detection may fail in some instances, but tracking may still be achievable which takes into account the location and appearance of the object in the previous frame. The key hurdles in real-time image and video processing applications are object tracking and motion detection. Some tracking algorithms are extremely fast because they perform a local search rather than a global search. Tracking algorithms such as meanshift, Regional Neural Network probabilistic data association, particle filter, nearest neighbor, Kalman filter and interactive multiple model (IMM) are available to estimate and predict the state of a system.

Augmented reality-computer vision combination for automatic fatigue crack detection and localization

Fatigue cracks in bridges are inspected visually by specialized bridge inspection professionals. Bridge inspectors conduct inspections in a limited amount of time, and at times small cracks may go unnoticed. Researchers have recently developed computer-based techniques to help overcome these issues. This study uses a computer vision algorithm combined with Augmented Reality (AR) to localize fatigue cracks during the visual inspection that otherwise may go unnoticed because of their size. The AR software utilizes a video processing algorithm for fatigue crack detection. Subsequently, the AR software generates a hologram using the algorithm’s detection result and anchors it over the crack’s location at the structure being inspected. The result of this new methodology is an automatic fatigue crack detection and localization AR software that provides holograms overlaid during the on-site visual inspection. This technique also provides the fatigue crack detection result in near-real-time. The method is verified using 2D and 3D benchmarks and a half-scale steel bridge girder specimen. The research team collected the feedback from bridge industry inspectors contextualized in real operations and their recommendations for integrating AR for inspections with the involvement of human resources and workforce development. This study is the first effort of holographic addressing the fatigue crack on the actual structure using AR headset.

Low-energy Pipelined Hardware Design for Approximate Medium Filter

Image and video processing algorithms are currently crucial for many applications. Hardware implementation of these algorithms provides higher speed for large computation applications. Removing noise is often a typical pre-processing step to enhance the results of later analysis and processing. Median filter is a typical nonlinear filter that is very commonly used for impulse noise elimination in digital image processing. This article suggests a low-energy median filter hardware design for battery-based hardware applications. An approximate solution with high accuracy is investigated to speed up the filtering operation, reduce the area, and consume less power/energy. Pipelining and parallelism are used to optimize the speed and power of this technique. Non-pipelined, two different pipelined structures, and two parallel architectures versions are designed. The design versions are implemented first with a Virtex-5 LX110T FPGA and then using the UMC 130nm standard cell ASIC technology. The selection and the even-odd sorting-based median filters are also implemented for an equitable comparison with the standard median filtering techniques. The suggested non-pipelined median filter design enhances the throughput 35% more than the highest investigated state of the art. The pipelining enhances the throughput to more than twice its value. Additionally, the parallel architecture decreases the area and the consumed power by around 40%. The simulation results reveal that one of the suggested designs significantly decreases the area, with the same speed as the fastest design in the literature, without noticeably degrading the accuracy, and a significant decrease in energy consumption by about 60%.

Current Technologies and Applications of Digital Image Processing

There has been significant advances in the field of image and video processing over the past few decades. The term "image processing" is used to describe multiple signal-processing methodologies where images (such as video or picture frames) serves as the input, resulting to another image or a collection of image-related parameters or features. The majority of methodologies to image processing include reducing the picture to a two-dimensional signal and processing it in the same way as any other signal. The term "video processing" on the other hand is used to describe a particular type of signal processing where video files or video streams are utilized as output or input signals. Video recorders, televisions, video codecs, digital versatile, disc players, and other devices all utilize video processing algorithms. This paper provides a survey of the components of Digital Image Processing (DIP) as well as the recent developments in Image Processing technology and DIP applications.

Разработка метода обработки видео, повышающего оценку метрики качества VMAF на основе её дистилляции

In this work, we consider the problem of creating a video preprocessing method that improves video’s quality score measured by the Video Multimethod Assessment Fusion (VMAF) metric. The paper describes a neural network method for automatic preprocessing of input video, operating in real time. Preprocessing is carried out by a deep neural network based on U-Net architecture. In the course of network training, a trained VMAF approximation is used. The paper describes ways of improving the quality of the final method, namely, adding neural network compression, using SSIM in the loss function, and filtering the training set. The final version of the method increases the VMAF score of the original video by an average of 18% after preprocessing. The developed method demonstrates the flaws of the VMAF quality assessment method that can be used by developers of video processing algorithms to improve the ratings of their methods during automatic comparison carried out using VMAF quality assessment method.

A video processing algorithm using temporal intuitionistic fuzzy sets

Video Processing has found enormous applications in recent times from security systems to interplanetary missions. In real-life situations, most of the videos are fuzzy/vague/uncertain. Intuitionistic fuzzy set (IFS) is one of the effective tools for handling uncertainty. Among many extensions of IFSs, temporal intuitionistic fuzzy sets (TIFSs) are very interesting as they are time-dependent. Hence, TIFSs are suitable to define a video, which is dynamic and hence depends on time-moment. In this way, this work introduces a novel VIdeo PROCessing (VIPROC) algorithm, using temporal intuitionistic fuzzy sets to enhance videos, which is first of its kind in existence. The comparison is made with fuzzy contrast intensification operation. VIPROC algorithm is designed using contrast intensification operation for video enhancement. The results are encouraging in comparison with the original test videos. The results are discussed taking into account the several frames of the test video. Further, the proposed algorithm can be applied/extended to engineering applications like motion tracking, traffic detection systems, real time videos captured through mobile (hand-held) devices, and so on. As no such algorithms are existing which use TIFSs to process a video, the authors got motivated to design and develop VIPROC algorithm.

Edge–Artificial Intelligence-Powered Parking Surveillance With Quantized Neural Networks

The rapid development of urbanization has raised challenges for existing parking facilities to serve increasing parking demand. Being constrained by the limited urban land resources for newly constructed parking facilities, improving the efficiency of existing parking infrastructures relies more on an advanced parking management strategy. Video-based parking surveillance technology, with abundant information, easy installation, and powerful algorithms, is the most popular method to provide real-time parking information as the required data feed for a parking management system. Meanwhile, deep learning-based algorithms gradually replace traditional computer vision methods for video processing since the detection accuracy has been increased considerably by learning fine-grained features. However, the excessive computational complexity of deep learning-based algorithms occupies considerable computational resources, which certainly hurts the entire system’s efficiency. Due to the limited computing power of edge devices, most parking surveillance systems deploy video processing algorithms on a server or cloud platform, which raises concerns about data transmission latency and central computation pressure. Deploying efficient algorithms on an edge-side device is a potential solution to solve these problems. This article proposes an edge computing parking occupancy detection system with a quantized deep learning model. Model quantization is employed to boost the inference speed while maintaining accuracy. In addition, knowledge distillation is applied to improve the quantized model’s training performance. Experiments are conducted to demonstrate the model’s superiority compared to state-of-the-art algorithms and the feasibility of edge computing. The proposed method can improve the accuracy and efficiency of parking surveillance systems. It is a systematic solution for obtaining parking information with limited computational resources.

713 The Application of Artificial Intelligence for Digital Imaging in the Operating Theatre: A Systematic Review and Narrative Synthesis

Abstract Introduction Promising applications of artificial intelligence (AI) in healthcare are emerging. This systematic review aims to identify and synthesise applications of digital-imaging AI in surgery and inform future work. Method Systematic database searches (Medline, Embase, CENTRAL) were undertaken. Studies concerning digital-imaging AI within the operating theatre were identified from title and abstract screening. Selection was further refined to identify video-based AI models with direct supportive output to the surgeon within the operating theatre. Results 48 studies were included. Studies spanned 13 specialty groupings, with n=42 utilising a pre-specified dataset and the remaining n=6 using AI with human participants. The most common field using AI was urology (n=9 studies). Applications were most commonly for navigation and visualisation support (n=26 studies across 10 surgical specialties) and AI-based intelligent detection systems, intended to identify and highlight useful surgical information using computer-vision pattern recognition (n=18 articles across n=6 specialties). Other applications included video-processing algorithms (n=3 studies across 2 specialties), and a novel imaging modality for visualising blood perfusion (n=1 study), proposing operating theatre-based application. High-performance models were identified across a range of pathologies. This manifested as minimal overlay errors and acceptable frame rates for navigation tools, and high diagnostic performance for detection systems (determined by area-under-the-receiver-operating-characteristic-curve, sensitivity/specificity, negative/positive predictive values). Conclusions There is evidence to suggest AI for intraoperative surgeon-support has potential, particularly through augmented-reality navigation and AI-enabled information awareness. Further research and optimisations are required to produce clinically robust models, which remain high-performance despite case variability. Such AI may support improved surgical access, efficiency, and outcomes.

The illegal parking score – Understanding and predicting the risk of parking illegalities in Lisbon based on spatiotemporal features

Illegal parking represents a costly problem for most cities as it leads to an increase in traffic congestion and emission of air pollutants, and decreases pedestrian, biking, and driving safety, making cities less clean, secure, and attractive to citizens and tourists. Most decision-support systems employed to deal with parking illegalities rely on cameras and video-processing algorithms to capture infractions in real-time. Despite being effective, their implementation is costly and challenging due to road environment conditions. On the other hand, studies that relay on spatiotemporal features to predict infractions can present a more efficient alternative, one that is less costly to implement and free of environment and spatial conditioning. In this work, we propose the Illegal Parking Score (IPS), a score of the conditional probability of illegal parking occurring in a road segment, based on spatiotemporal conditions, and able to distinguish between illegality types. The IPS is calculated for the Lisbon Municipality, in Portugal, and it is supported by a Light Gradient Boosting Machine model that allows for IPS prediction for unseen conditions. Likewise, we propose the IPS Simulator, a simulation tool that allows for users to infer the IPS by defining spatiotemporal conditions. This system will be deployed in the Lisbon City Council and provides responsible authorities with a tool to support their daily operations and promote sustainable transport and demand planning, by identifying and monitoring critical zones and by aiding in the design and gauge of parking regulation.

Review of the Current Technologies and Applications of Digital Image Processing

There has been significant advances in the field of image and video processing over the past few decades. The term "image processing" is used to describe multiple signal-processing methodologies where images (such as video or picture frames) serves as the input, resulting to another image or a collection of image-related parameters or features. The majority of methodologies to image processing include reducing the picture to a two-dimensional signal and processing it in the same way as any other signal. The term "video processing" on the other hand is used to describe a particular type of signal processing where video files or video streams are utilized as output or input signals. Video recorders, televisions, video codecs, digital versatile, disc players, and other devices all utilize video processing algorithms. This paper provides a survey of the components of Digital Image Processing (DIP) as well as the recent developments in Image Processing technology and DIP applications.

Object Detection and Recognition in Real Time Using Deep Learning for Visually Impaired People

This paper aims at combining object detection at real time and recognition with suitable deep learning methods in order to detect and recognize objects position as well as the names of multiple objects detected by the camera using an object detector algorithm. This is to aid the visually impaired user without the help of any other person. The image and video processing algorithms were designed to take real-time inputs from the camera, Deep Neural Networks were used to predict the objects and uses Google’s famous Text-To-Speech (GTTS) API module for the anticipated voice output precisely detecting and recognizing the category or class of objects and locations contained. Our best result shows that the system recognizes 91 categories of outdoor objects and produces the output in speech i.e. in an audio format even when a reduced amount of spectral information from the data is available.

Modular counting management system for mall parking services

Motorcycle customers are a dominant consumer group in developing countries or regions around the world. Although motorcyclist parking is very important to the management and operation of the mall, the issue is rarely concerned. The existing parking management systems are mostly used for vehicle delivery management. At the same time, if the traditional parking management system needs to be updated, the large-scale construction of the entrance and exit is usually required, which will increase the construction cost and affect customer satisfaction, leading to the overall profit loss of the mall. Therefore, this paper develops a new system, named Modular Counting Management System (MCMS), which utilizes CCD video and digital image processing (DIP) and counting algorithms for real-time monitoring without resetting the entrance device. The system can be implemented easily, provide sufficient monitoring management information to effectively improve the utilization rate of motorcycle parking areas and combine with other identification systems for the mall. This study has three main contributions: (1) A MCMS has been developed and deployed for a large shopping mall in Taiwan to improve the satisfaction of motorcyclists and increase profits; (2) The proposed MCMS provides adequate monitoring management information to effectively improve the usage rate of motorcycle parking areas; and (3) A survey of motorcyclist opinion showed an 82% satisfaction rate from a total of 479 effective samples and revenues increased 13.8% in the year following the implementation of this system. These have proven the system's contribution to practice and theory.

Live Free-Viewpoint Video in Immersive Media Production Over 5G Networks

5G is a key tool for the cloud-based wireless production of audio-visual contents. By providing higher throughput and lower latency than previous mobile network technologies, as well as flexible allocation of network resources, it enables that some content production pipelines, which are currently implemented using wired connectivity, are provided on top of a 5G network. This brings down production costs, reduces environmental impact and increases operational efficiency. In particular, live immersive production services, such as free-viewpoint video (FVV), can be provided in a much more efficient way. In this paper, we address the challenge of producing and streaming FVV services in real time through 5G networks. We have adapted a state-of-the-art FVV system to integrate it within a 5G architecture, using three key technological enablers: mmWave radio access network to support the uplink traffic requirements from FVV cameras, multi-access edge computing to run video processing algorithms with minimum latency, and end-to-end slicing to guarantee a sufficient quality of service (QoS) within the production pipeline. We have built a field trial over the production network of a telecommunication operator, including a mmWave pilot deployment, edge cloud processing, and remote content production, involving three different locations across Spain. We have measured the key performance indicators at the relevant parts of our trial deployment, showing that, with existing 5G technology, it is possible to achieve live FVV production, although with some limitations. We have also analyzed such limitations, obtaining some insights on how the next generation of 5G networks can overcome them to achieve higher quality of experience (QoE).

Low Power Intrusion Detection Using Multi-Level Sensor Authentication

Surveillance systems be connected with very less cost and less power IR and PIR sensors, which are more capable to find the false alarms and small inference. Wireless smart cameras become an appropriate solution for the upcoming challenges like the hardware capabilities in terms of low ­ power utilization and high imaging performance and high memory footprint all are very big tasks for the wireless smart cameras. For this reason, wireless surveillance systems still involve a significant aggregate of research in different areas such as mote architectures, video processing algorithms, power management, energy harvesting and distributed engine. Here we evolve a solution for this problem through multilevel sensing architecture. The proposed system to be find the energy interference in the first level. If any unwanted event, it will be occurring at the moment the additional authentication unit become initiate and activities. In this stage, PIR sensor that detects the traces of the event. If the PIR sensor detects the same, it authenticates the event and switches ON the wireless camera. This system has multiple advantages like reduced power consumption, improved event detection accuracy, longer life span and enhanced information clarity.