Video Processing Pipeline Research Articles

Instinctive Detection of Accident Occurrence using Numerous Machine Learning Techniques with Comparative Study

Objectives: The research work aims to develop an automated detection system that uses video captures to identify roadside accidents or significant events. By alerting nearby hospitals and emergency services, the system reduces response times and potentially saves lives. The system’s integration with existing emergency response systems ensures prompt assistance to those in need. Methods: The video processing pipeline begins by converting video files into frames for analysis and feature extraction. These features serve as inputs for classification algorithms such as Random Forest, SVM, and KNN. The model’s performance is evaluated using a training set and unseen test data, with the predicted classifications compared against the ground truth labels. Findings: Among the tested classification algorithms, the Random Forest algorithm achieved the highest accuracy. Using 128 frames for analysis provided more comprehensive information, yielding a 96% accuracy rate. This combination proves to be a powerful tool in classification tasks, providing reliable and accurate outputs. Novelty: Machine learning algorithms are instrumental in automating accident detection from video captures. They analyse video footage to identify accidents and promptly alert relevant authorities. This technology can also dispatch emergency messages to nearby hospitals, ensuring quick assistance. The consideration of different frame counts in classification improves accuracy by capturing critical moments and patterns. Machine learning algorithms applied in this work significantly enhance emergency response, reduce response times, and potentially save lives. Keywords: ­ Emergency Services, Accidents, Feature Extraction, Random Forest, SVM, KNN

A Video Segmentation Pipeline for Assessing Changes in Pupil Response to Light After Cannabis Consumption

Due to long-standing federal restrictions on cannabis-related research, the implications of cannabis legalization on traffic and occupational safety are understudied. Accordingly, there is a need for objective and validated measures of acute cannabis impairment that may be applied in public safety and occupational settings. Pupillary response to light may offer an avenue for detection that outperforms typical sobriety tests and tetrahydrocannabinol concentrations. We developed a video processing and analysis pipeline that extracts pupil sizes during a light stimulus test administered with goggles utilizing infrared videography. The analysis compared pupil size trajectories in response to a light for those with occasional, daily, and no cannabis use before and after smoking. Pupils were segmented using a combination of image pre-processing techniques and segmentation algorithms which were validated using manually segmented data and found to achieve 99% precision and 94% F-score. Features extracted from the pupil size trajectories captured pupil constriction and rebound dilation and were analyzed using generalized estimating equations. We find that acute cannabis use results in less pupil constriction and slower pupil rebound dilation in the light stimulus test.

S3-VAE: A novel Supervised-Source-Separation Variational AutoEncoder algorithm to discriminate tumor cell lines in time-lapse microscopy images

The derivation of input–output relationships in deep learning architectures is mostly a black-box process, in which uninformative or confounding factors might bias the classification results, with no clue for the user. However, the analysis of living cells requires extracting latent space representations with interpretable meaning, which can be investigated not only for classification but mainly for understanding purposes. Using the properties of variational autoencoders in deriving unsupervised regular space representations, we propose a novel Supervised-Source-Separation Variational Autoencoder (S3-VAE) algorithm capable of guiding the encoding process of images toward a more effective class-separability. The scope has been reached, under the assumption of full-covariance Gaussian posterior, by introducing a term in the cost function that forces class-dependent sources of variations toward a one-hot encoding. The proposed approach has been designed for an automatic platform for single-cell biological investigations based on time-lapse microscopy, in which a dedicated video processing pipeline has been designed, as an additional contribution, to reduce the effects of confounding factors on the S3-VAE representations. The results obtained in a classification scenario with artificially generated phantom cells and with experimental data of human prostate cells, in non-neoplastic, neoplastic and metastatic neoplastic conditions, show that S3-VAE-based latent representations allow for investigation and recognition of the morphological differences among classes. Comparative analysis with other benchmark deep learning architectures demonstrates the effectiveness of the proposed approach and shows that S3-VAE is capable of achieving high performance in discriminating cell lines based on phenotypic variations.

Equine Kinematic Gait Analysis Using Stereo Videography and Deep Learning: Stride Length and Stance Duration Estimation

Highlights Stereo machine vision and deep learning techniques were investigated for infield equine kinematic gait analysis. The proposed pipeline tracks equine body landmarks in 3D space and estimates stride length and stance duration. The system can serve as a cost-effective, rapid, and easy-to-use tool for equine locomotion research. Abstract. Equine kinematic gait analysis (EKGA) currently requires a complicated, expensive, and labor-intensive procedure for equine locomotion research. An automated stereo video processing pipeline was developed and evaluated for measuring equine biomechanical parameters. Using stereo videos of 40 different walking horses, a DeepLabCut (DLC) model was trained to detect body landmarks in individual frames. With an autoregressive integrated moving average filter, the landmark detection had a root mean square error of 5.14 pixels and a mean absolute error of 4.87 pixels. As a case study, methods were developed to extract stride length (SL) and stance duration (SD). Individual hoof gait phase detection was achieved using a fine-tuned Faster R-CNN model and a mode filter, yielding precision and recall values of 0.83 and 0.95, respectively. The semi-global block matching (SGBM) algorithm was used to estimate depth maps, and the accuracy was assessed by comparing head length estimation with infield measurements. A Bland-Altman analysis for DLC-detected head length in combination with SGBM-based 3D reconstruction yielded a bias of -0.014 m with upper and lower limits of agreement (LoAs) of 0.03 m and -0.061 m, respectively. Furthermore, Bland-Altman analyses on SD and SL when compared to image-level manual measurements showed biases of -0.02 sec and -0.042 m, respectively. The corresponding LoAs were (0.01907 sec, -0.24 sec) for SD and (0.04 m, -0.12 m) for SL. The proposed method showed promising potential in performing EKGA in an automated, cost-effective, and rapid manner under field conditions. Keywords: 3D Reconstruction, Animal Pose Estimation, Deep Learning, Equine Kinematic Gait Analysis, Stereo Matching.

CharmSeeker: Automated Pipeline Configuration for Serverless Video Processing

Video processing plays an essential role in a wide range of cloud-based applications. It typically involves multiple pipelined stages, which well fits the latest fine-grained serverless computing paradigm if properly configured to match the cost and delay constraints of video. Existing configuration tools, however, are primarily developed for traditional virtual machine clusters with general workloads. This paper presents CharmSeeker, an automated configuration tuning tool for serverless video processing pipelines. We first carefully examine the key steps and the performance bottlenecks for video processing over modern serverless platforms. Then, we identify the configuration space for processing pipelines and leverage a carefully designed Sequential Bayesian Optimization search scheme to identify promising configurations. We further address the practical challenges toward integrating our solution into real-world systems and develop a prototype with AWS Lambda. Evaluation results show that CharmSeeker can find out the optimal or near-optimal configurations that improve the relative processing time up to 408.77%. It is also more robust and scalable to various video processing pipelines compared with state-of-the-art solutions.

Reviewing ViBe, a Popular Background Subtraction Algorithm for Real-Time Applications

Background subtraction or foreground segmentation is commonly the first step in video processing pipelines, where foreground objects and moving entities are detected for further analysis. Background subtraction is still an open problem in computer vision due to the wide range of possible scenarios and challenging cases, such as weather conditions, illumination variation, dynamic backgrounds or low frame-rates. In this article, we review a well-known general algorithm for background subtraction named visual background extractor or ViBe. More than 10 years after its publication, ViBe is still widely used due to its simplicity and low computational load, as its authors claim it can process up to 200 fps. We analyze the algorithm's mechanism to segment foreground objects, comment on its strengths and drawbacks, and describe an effective variant of the method to quickly dissipate ghost effects using 3-frame difference. Furthermore, we provide an easy-to-use demo that allows to quickly test the performance of ViBe on custom data, modify its parameters and download the results. **This is an MLBriefs article, the source code has not been reviewed!**<br> The ViBe algorithm is covered by several patents and its code is not provided with this publication. <a href="https://orbi.uliege.be/handle/2268/145853">Its code is only shared for research purposes.</a><br>

Towards Automating Retinoscopy for Refractive Error Diagnosis

Refractive error is the most common eye disorder and is the key cause behind correctable visual impairment, responsible for nearly 80% of the visual impairment in the US. Refractive error can be diagnosed using multiple methods, including subjective refraction, retinoscopy, and autorefractors. Although subjective refraction is the gold standard, it requires cooperation from the patient and hence is not suitable for infants, young children, and developmentally delayed adults. Retinoscopy is an objective refraction method that does not require any input from the patient. However, retinoscopy requires a lens kit and a trained examiner, which limits its use for mass screening. In this work, we automate retinoscopy by attaching a smartphone to a retinoscope and recording retinoscopic videos with the patient wearing a custom pair of paper frames. We develop a video processing pipeline that takes retinoscopic videos as input and estimates the net refractive error based on our proposed extension of the retinoscopy mathematical model. Our system alleviates the need for a lens kit and can be performed by an untrained examiner. In a clinical trial with 185 eyes, we achieved a sensitivity of 91.0% and specificity of 74.0% on refractive error diagnosis. Moreover, the mean absolute error of our approach was 0.75±0.67D on net refractive error estimation compared to subjective refraction measurements. Our results indicate that our approach has the potential to be used as a retinoscopy-based refractive error screening tool in real-world medical settings.

63‐4: A Novel Real‐Time Full‐Color 3D Holographic (Diffractive) Video Capture, Processing, and Transmission Pipeline Using Off‐the‐Shelf Hardware

This paper details the world's first live 3D holographic (diffractive) video call using off‐the‐shelf hardware. We introduce a novel pipeline that facilitates the capture, processing, and transmission of RGBZ data, using an iPhone for image and depth capture with VividQ's SDK for hologram generation and hardware for display.

Objective Video Quality Assessment Method for Face Recognition Tasks

Nowadays, there are many metrics for overall Quality of Experience (QoE), both those with Full Reference (FR), such as Peak Signal-to-Noise Ratio (PSNR) or Structural Similarity (SSIM), and those with No Reference (NR), such as Video Quality Indicators (VQI), which are successfully used in video processing systems to evaluate videos whose quality is degraded by different processing scenarios. However, they are not suitable for video sequences used for recognition tasks (Target Recognition Videos, TRV). Therefore, correctly estimating the performance of the video processing pipeline in both manual and Computer Vision (CV) recognition tasks is still a major research challenge. There is a need for objective methods to evaluate video quality for recognition tasks. In response to this need, we show in this paper that it is possible to develop the new concept of an objective model for evaluating video quality for face recognition tasks. The model is trained, tested and validated on a representative set of image sequences. The set of degradation scenarios is based on the model of a digital camera and how the luminous flux reflected from the scene eventually becomes a digital image. The resulting degraded images are evaluated using a CV library for face recognition as well as VQI. The measured accuracy of a model, expressed as the value of the F-measure parameter, is 0.87.

Video-Based Analysis and Reporting of Riding Behavior in Cyclocross Segments.

Video-based trajectory analysis might be rather well discussed in sports, such as soccer or basketball, but in cycling, this is far less common. In this paper, a video processing pipeline to extract riding lines in cyclocross races is presented. The pipeline consists of a stepwise analysis process to extract riding behavior from a region (i.e., the fence) in a video camera feed. In the first step, the riders are identified by an Alphapose skeleton detector and tracked with a spatiotemporally aware pose tracker. Next, each detected pose is enriched with additional meta-information, such as rider modus (e.g., sitting on the saddle or standing on the pedals) and detected team (based on the worn jerseys). Finally, a post-processor brings all the information together and proposes ride lines with meta-information for the riders in the fence. The presented methodology can provide interesting insights, such as intra-athlete ride line clustering, anomaly detection, and detailed breakdowns of riding and running durations within the segment. Such detailed rider info can be very valuable for performance analysis, storytelling, and automatic summarization.

Deep authoring - an AI Tool set for creating immersive MultiMedia experiences

We introduce a fully automated 360° video processing pipeline using a hierarchical combination of Artificial Intelligence (AI) modules to create immersive volumetric XR experiences. Two critical productions tasks (person segmentation and depth estimation) are addressed with a parallel Deep Neural Network (DNN) pipeline that combines instance segmentation, person detection, pose estimation, camera stabilization, neural tracking, 3D face detection, hair masking, and monocular 360° depth computation in a single and robust tool set. To facilitate the rapid uptake of these techniques we provide a detailed review of AI-based methods to address these problems (complete with links to recommended open source implementations) as well as references to existing authoring tools in the market. Our key contributions include a method to create semi-synthetic data sets for data auto-augmentation and using this technique to generate over 3.8 m images as part of a concise evaluation and subsequent retraining of DNNs for person detection tasks. Furthermore, we apply the same techniques to develop a spherical DNN for monocular depth estimation with a Free Viewpoint Video (FVV) capture system and a novel method to generate 3D human shapes and pose mannequins for training. To evaluate the performance of our AI authoring tool set we address four challenging production tasks and demonstrate the practical use of our solution with videos showing processed output.

Enabling Dynamic System Integration on Maxeler HLS Platforms

High Level Synthesis (HLS) tools enable application domain experts to implement applications and algorithms on FPGAs. The majority of present FPGA applications is following a stream processing model which is almost entirely implemented statically and not exploiting the benefits enabled by partial reconfiguration. In this paper, we propose a generic approach for implementing and using partial reconfiguration through an HLS design flow for Maxeler platforms. Our flow extracts HLS generated HDL code from the Maxeler compilation process in order to implement a static FPGA infrastructure as well as run-time reconfigurable stream processing modules. As a distinct feature, our infrastructure can accommodate multiple partial modules in a pipeline daisy-chained manner, which aligns directly to Maxeler’s dataflow programming paradigm. The benefits of the proposed flow are demonstrated by a case study of a dynamically reconfigurable video processing pipeline delivering 6.4GB/s throughput.

GPU acceleration of NL-means, BM3D and VBM3D

Denoising is an essential part of any image- or video-processing pipeline. Unfortunately, due to time-processing constraints, many pipelines do not consider the use of modern denoisers. These algorithms have only CPU implementations or suboptimal GPU implementations. We propose a new efficient GPU implementation of NL-means and BM3D, and, to our knowledge, the first GPU implementation of the video-denoising algorithm VBM3D. The performance of these implementations enable their use in real-time scenarios.

Unsupervised learning-based long-term superpixel tracking

Finding correspondences between structural entities decomposing images is of high interest for computer vision applications. In particular, we analyze how to accurately track superpixels - visual primitives generated by aggregating adjacent pixels sharing similar characteristics - over extended time periods relying on unsupervised learning and temporal integration. A two-step video processing pipeline dedicated to long-term superpixel tracking is proposed. First, unsupervised learning-based superpixel matching provides correspondences between consecutive and distant frames using new context-rich features extended from greyscale to multi-channel and forward-backward consistency constraints. Resulting elementary matches are then combined along multi-step paths running through the whole sequence with various inter-frame distances. This produces a large set of candidate long-term superpixel pairings upon which majority voting is performed. Video object tracking experiments demonstrate the accuracy of our elementary estimator against state-of-the-art methods and proves the ability of multi-step integration to provide accurate long-term superpixel matches compared to usual direct and sequential integration.

A video processing pipeline for intraoperative analysis of cerebral blood flow

ABSTRACT In this paper, we propose a pipeline for non-invasive, high-resolution analysis of cerebral blood flow. We use a 3-CMOS HD-Videocamera to record video samples that are processed by a stabilization and colour amplification pipeline to visualize changes caused by the blood pressure cycle. To our knowledge, this is the first description of a non-invasive, only software-based algorithm which comprises information about brain perfusion.

Frame-based Programming, Stream-Based Processing for Medical Image Processing Applications

This paper presents and evaluates an approach to deploy image and video processing pipelines that are developed frame-oriented on a hardware platform that is stream-oriented, such as an FPGA. First, this calls for a specialized streaming memory hierarchy and accompanying software framework that transparently moves image segments between stages in the image processing pipeline. Second, we use softcore VLIW processors, that are targetable by a C compiler and have hardware debugging capabilities, to evaluate and debug the software before moving to a High-Level Synthesis flow. The algorithm development phase, including debugging and optimizing on the target platform, is often a very time consuming step in the development of a new product. Our proposed platform allows both software developers and hardware designers to test iterations in a matter of seconds (compilation time) instead of hours (synthesis or circuit simulation time).

Color HDR video processing architecture for smart camera

This paper presents a novel FPGA architecture of high dynamic range (HDR) video processing pipeline, based on the capturing of a sequence of differently exposed images. An acquisition process enabling multi-exposure HDR as well as fast implementation of local tone mapping operator involving bilateral filtering is proposed. The HDR acquisition process is enhanced by the application of novel deghosting method, which is dedicated for hardware implementation and proposed in this paper. The hardware processing pipeline is designed with regards to efficiency and performance and the calculations are performed in fixed point arithmetic. The pipeline is suitable for programmable hardware (FPGA—Field Programmable Gate Arrays) implementation and it achieves real-time performance on full HD HDR video which overcomes state-of-the-art solutions that use local tone mapping and deghosting algorithm.

Unmanned Aerial Vehicles (UAVs) and Artificial Intelligence Revolutionizing Wildlife Monitoring and Conservation

Surveying threatened and invasive species to obtain accurate population estimates is an important but challenging task that requires a considerable investment in time and resources. Estimates using existing ground-based monitoring techniques, such as camera traps and surveys performed on foot, are known to be resource intensive, potentially inaccurate and imprecise, and difficult to validate. Recent developments in unmanned aerial vehicles (UAV), artificial intelligence and miniaturized thermal imaging systems represent a new opportunity for wildlife experts to inexpensively survey relatively large areas. The system presented in this paper includes thermal image acquisition as well as a video processing pipeline to perform object detection, classification and tracking of wildlife in forest or open areas. The system is tested on thermal video data from ground based and test flight footage, and is found to be able to detect all the target wildlife located in the surveyed area. The system is flexible in that the user can readily define the types of objects to classify and the object characteristics that should be considered during classification.

A Design Approach to Creating Scalable Video Processing Systems beyond UHDTV1 on an FPGA

An unquenchable end-user thirst for enhanced video quality results in ever-scaling video frame size and frame rate requirements. As we move from UHDTV1 to UHDTV2 and 120 frames/sec to 300 frames/sec, inevitably the computational complexity of video processing systems required to consume, process, and deliver video content increases. The need for solutions to support combinations of frame sizes and rates, as well as future increments, emphasizes the need for system scalability. The computational complexity and scalability requirements pose exciting challenges for field-programmable gate array (FPGA) implementation of video processing pipelines. This paper presents implementation techniques and methodologies to overcome these challenges. We specifically concentrate on architectures whereby the input per-pixel video sample rate exceeds the system clock rate. Novel results include classifying pixel processing orders and presenting a component-based design approach for future-proofing video processing solutions against an ever-scaling computational complexity requirement. Resource and memory bandwidth requirements of such systems are also analyzed and trends are presented.

Multilevel Depth and Image Fusion for Human Activity Detection

Recognizing complex human activities usually requires the detection and modeling of individual visual features and the interactions between them. Current methods only rely on the visual features extracted from 2-D images, and therefore often lead to unreliable salient visual feature detection and inaccurate modeling of the interaction context between individual features. In this paper, we show that these problems can be addressed by combining data from a conventional camera and a depth sensor (e.g., Microsoft Kinect). We propose a novel complex activity recognition and localization framework that effectively fuses information from both grayscale and depth image channels at multiple levels of the video processing pipeline. In the individual visual feature detection level, depth-based filters are applied to the detected human/object rectangles to remove false detections. In the next level of interaction modeling, 3-D spatial and temporal contexts among human subjects or objects are extracted by integrating information from both grayscale and depth images. Depth information is also utilized to distinguish different types of indoor scenes. Finally, a latent structural model is developed to integrate the information from multiple levels of video processing for an activity detection. Extensive experiments on two activity recognition benchmarks (one with depth information) and a challenging grayscale + depth human activity database that contains complex interactions between human-human, human-object, and human-surroundings demonstrate the effectiveness of the proposed multilevel grayscale + depth fusion scheme. Higher recognition and localization accuracies are obtained relative to the previous methods.