Real Video Sequences Research Articles

APPLICATION OF MACHINE LEARNING FOR RECOGNIZING SURFACE WELDING DEFECTS IN VIDEO SEQUENCES

The paper offers a solution to the problem of detecting and recognizing surface defects in welded joints that appear during tungsten inert gas welding of metal edges. This problem belongs to the machine vision. Welding of stainless-steel edges is carried out automatically on the pipe production line. Therefore, frames of video sequences are investigated. Images of some welding defects are shown in the paper. An algorithm proposed by the authors is used to detect welding defects in the video sequence frames, the efficiency of which has been confirmed experimentally. The problem solution of welding defects recognition is based on the use of traditional machine learning methods: support vector machine and artificial neural network. To build classification models, a labeled dataset containing automatically extracted texture features from the areas of welding defects detected in the video sequences was created. An analysis was performed to identify the strength of the correlation of texture features between each other and the dependent variable in the dataset for dimensionality reduction of the feature vector. The models were trained and tested on datasets with different numbers of features. The quality of the classification models was evaluated based on the accuracy metric values. The best results were achieved by the classifier built using the support vector machine with a chi-square kernel on a training sample with two features. The build models allow automatic recognition of such welding defects as lack of fusion and metal oxidation. The computational experiments with real video sequences obtained with a digital camera confirmed the possibility of using the proposed solution for recognizing surface welding defects in the process of manufacturing stainless steel pipes.

Stabilization of turbulence-degraded video using patch-based reference frame

Imaging over long distances in the atmosphere can result in significant degradation of the acquired videos due to atmospheric turbulence. This degradation can include problems such as blurring and geometric distortion, and it is challenging to correct these distortions because of their randomness in nature. To address this, a new method for improving the geometrical quality of video sequences of remote stationary scenes is introduced in this paper. The method uses a patch-based approach to obtain a reference frame from the distorted video sequence, selecting the best quality patches from different frames. A window-based image registration method is then used to estimate the geometrical shifts of the pixels, which are used to restore a high-quality frame. The proposed method is compared with two similar state-of-the-art video stabilization methods using experiments on both synthetic and real video sequences. Results demonstrate that the patch-based method outperforms the other methods in terms of accuracy.

Application of improved you only look once model in road traffic monitoring system

<span lang="EN-US">The present research focuses on developing an intelligent traffic management solution for tracking the vehicles on roads. Our proposed work focuses on a much better you only look once (YOLOv4) traffic monitoring system that uses the CSPDarknet53 architecture as its foundation. Deep-sort learning methodology for vehicle multi-target detection from traffic video is also part of our research study. We have included features like the Kalman filter, which estimates unknown objects and can track moving targets. Hungarian techniques identify the correct frame for the object. We are using enhanced object detection network design and new data augmentation techniques with YOLOv4, which ultimately aids in traffic monitoring. Until recently, object identification models could either perform quickly or draw conclusions quickly. This was a big improvement, as YOLOv4 has an astoundingly good performance for a very high frames per second (FPS). The current study is focused on developing an intelligent video surveillance-based vehicle tracking system that tracks the vehicles using a neural network, image-based tracking, and YOLOv4. Real video sequences of road traffic are used to test the effectiveness of the method that has been suggested in the research. Through simulations, it is demonstrated that the suggested technique significantly increases graphics processing unit (GPU) speed and FSP as compared to baseline algorithms.</span>

A novel video magnification method for revealing subtle signals based on adaptive Laplacian pyramid

Numerous difficult to perceive minor signals, such as minute color variations and modest movements, carry crucial information. The ability to see small signals in video has been demonstrated to be greatly aided by video magnification techniques. The Eulerian video magnification (EVM) method produces excellent results when magnifying minute colors and motions; but, it is susceptible to artifacts and outcomes that are blurry, especially when bigger magnifications are used. In this research, we propose a novel video magnification method based on adaptive Laplacian pyramid to achieve non-contact optical measurements. Experiments were conducted on real video sequences containing biological or mechanical motion signals, the results demonstrated that our approach achieves video amplification and extraction of subtle signals, supports larger amplification factors, and obtains sharper video quality with reduced artifacts and blurring.

A Novel Framework to Generate Synthetic Video for Foreground Detection in Highway Surveillance Scenarios

Foreground detection (FD) plays an important role in the domain of video surveillance for highway. The design of advanced FD algorithms requires large-scale and diverse video dataset. However, collecting and labeling real dataset is still time-consuming, labor-intensive, and highly subjective. To address this issue, we first use computer graphics (CG) to clone real highway scenarios (HS) and generate synthetic multi-challenge video datasets, called “Synthetic-HS (CG)”, automatically labeled with accurate pixel-level ground truth. The Synthetic-HS (CG) dataset contains eight imaging condition sequences for computer vision research. Then, we design an image translation (IT) model that translates source domain (Synthetic-HS (CG)) to target domain (real). This model uses skip connections and attention module to generate realistic synthetic images “Synthetic-HS (IT)”. We use publicly available Synthetic-HS in combination with the corresponding real video sequence to conduct experiments. The experiment results suggest that: 1) The Synthetic-HS (CG) dataset enables us to provide precise quantitative evaluation of the drawbacks of foreground detection methods 2) The realistic Synthetic-HS (IT) images can be used to promote the visual perception in highway video surveillance.

Self scale estimation of the tracking window merged with adaptive particle filter tracker

Tracking a mobile object is one of the important topics in pattern recognition, but style has some obstacles. A Reliable tracking system must adjust their tracking windows in real time according to appearance changes of the tracked object. Furthermore, it has to deal with many challenges when one or multiple objects need to be tracked, for instance when the target is partially or fully occluded, background clutter, or even some target region is blurred. In this paper, we will present a novel approach for a single object tracking that combines particle filter algorithm and kernel distribution that update its tracking window according to object scale changes, whose name is multi-scale adaptive particle filter tracker. We will demonstrate that the use of particle filter combined with kernel distribution inside the resampling process will provide more accurate object localization within a research area. Furthermore, its average error for target localization was significantly lower than 21.37 pixels as the mean value. We have conducted several experiments on real video sequences and compared acquired results to other existing state of the art trackers to demonstrate the effectiveness of the multi-scale adaptive particle filter tracker.

Shapes of You? Investigating the Acceptance of Video-Based AAL Technologies Applying Different Visualization Modes.

An aged population, increasing care needs, and a lack of (in)formal caregivers represent major challenges for our society today. Addressing these challenges fuels efforts and developments in innovative technologies leading to various existing AAL applications aiming at improving autonomy, independence, and security in older age. Here, the usage of video-based AAL technologies is promising as detailed information can be obtained and analyzed. Simultaneously, this type of technology is strongly connected with privacy concerns due to fears of unauthorized data access or inappropriate use of recorded data potentially resulting in rejection and non-use of the applications. As privacy-preserving visualizations of video data can diminish those concerns, this empirical study examines the acceptance and privacy perceptions of video-based AAL technology applying different visualization modes for privacy preservation (n = 161). These visualization modes differed in their degrees of visibility and identifiability, covering different levels of privacy preservation (low level: "Blurred" mode; medium level: "Pixel" and "Grey" modes; high level: "Avatar" mode) and are specifically evaluated based on realistic video sequences. The results of our study indicate a rather low acceptance of video-based AAL technology in general. From the diverse visualization modes, the "Avatar" mode is most preferred as it is perceived as best suitable to protect and preserve the users' privacy. Beyond that, distinct clusters of future users were identified differing in their technology evaluation as well as in individual characteristics (i.e., privacy perception, technology commitment). The findings support the understanding of potential users' needs for a successful future design, development, and implementation of video-based, but still privacy-preserving AAL technology.

A Visual Approach to Measure Cloth-Body and Cloth-Cloth Friction.

Measuring contact friction in soft-bodies usually requires a specialised physics bench and a tedious acquisition protocol. This makes the prospect of a purely non-invasive, video-based measurement technique particularly attractive. Previous works have shown that such a video-based estimation is feasible for material parameters using deep learning, but this has never been applied to the friction estimation problem which results in even more subtle visual variations. Because acquiring a large dataset for this problem is impractical, generating it from simulation is the obvious alternative. However, this requires the use of a frictional contact simulator whose results are not only visually plausible, but physically-correct enough to match observations made at the macroscopic scale. In this paper, which is an extended version of our former work A. H. Rasheed, V. Romero, F. Bertails-Descoubes, S. Wuhrer, J.-S. Franco, and A Lazarus, "Learning to measure the static friction coefficient in cloth contact," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2020, pp. 9909-9918, we propose to our knowledge the first non-invasive measurement network and adjoining synthetic training dataset for estimating cloth friction at contact, for both cloth-hard body and cloth-cloth contacts. To this end we build a protocol for validating and calibrating a state-of-the-art frictional contact simulator, in order to produce a reliable dataset. We furthermore show that without our careful calibration procedure, the training fails to provide accurate estimation results on real data. We present extensive results on a large acquired test set of several hundred real video sequences of cloth in friction, which validates the proposed protocol and its accuracy.

Single Image Video Prediction with Auto-Regressive GANs.

In this paper, we introduce an approach for future frames prediction based on a single input image. Our method is able to generate an entire video sequence based on the information contained in the input frame. We adopt an autoregressive approach in our generation process, i.e., the output from each time step is fed as the input to the next step. Unlike other video prediction methods that use “one shot” generation, our method is able to preserve much more details from the input image, while also capturing the critical pixel-level changes between the frames. We overcome the problem of generation quality degradation by introducing a “complementary mask” module in our architecture, and we show that this allows the model to only focus on the generation of the pixels that need to be changed, and to reuse those that should remain static from its previous frame. We empirically validate our methods against various video prediction models on the UT Dallas Dataset, and show that our approach is able to generate high quality realistic video sequences from one static input image. In addition, we also validate the robustness of our method by testing a pre-trained model on the unseen ADFES facial expression dataset. We also provide qualitative results of our model tested on a human action dataset: The Weizmann Action database.

A Convolutional Neural Network for Nonrigid Structure from Motion

In this study, we propose a reconstruction and optimization neural network (RONN), a novel neural network for nonrigid structure from motion, which is completed by an unsupervised convolution neural network. Compared with the traditional method for directly solving 3D structures, our model focuses on depth information that is lost owing to projection. This mathematical model is developed using a convolutional neural network with three modules for integration, reconstruction, and optimization, as well as two prior-free loss functions. The proposed RONN achieves competitive accuracy on several tested sequences and high visual quality of various real video sequences.

Deep Learning-Based Object Detection and Scene Perception under Bad Weather Conditions

Large cities’ expanding populations are causing traffic congestion. The maintenance of the city’s road network necessitates ongoing monitoring, growth, and modernization. An intelligent vehicle detection solution is necessary to address road traffic concerns with the advancement of automatic cars. The identification and tracking vehicles on roads and highways are part of intelligent traffic monitoring while driving. In this paper, we have presented how You Only Look Once (YOLO) v5 model may be used to identify cars, traffic lights, and pedestrians in various weather situations, allowing for real-time identification in a typical vehicular environment. In an ordinary or autonomous environment, object detection may be affected by bad weather conditions. Bad weather may make driving dangerous in various ways, whether due to freezing roadways or the illusion of low fog. In this study, we used YOLOv5 model to recognize objects from street-level recordings for rainy and regular weather scenarios on 11 distinct classes of vehicles (car, truck, bike), pedestrians, and traffic signals (red, green, yellow). We utilized freely available Roboflow datasets to train the proposed system. Furthermore, we used real video sequences of road traffic to evaluate the proposed system’s performance. The study results revealed that the suggested approach could recognize cars, trucks, and other roadside items in various circumstances with acceptable results.

Decomposition of Longitudinal Deformations via Beltrami Descriptors

We present a mathematical model to decompose a longitudinal deformation into normal and abnormal components. The goal is to detect and extract subtle abnormal deformation from periodic motions in a video sequence. It has important applications in medical image analysis. To achieve this goal, we consider a representation of the longitudinal deformation, called the Beltrami descriptor, based on quasiconformal theories. The Beltrami descriptor is a complex-valued matrix. Each longitudinal deformation is associated to a Beltrami descriptor and vice versa. To decompose the longitudinal deformation, we propose to carry out the low rank and sparse decomposition of the Beltrami descriptor. The low rank component corresponds to the periodic motions, whereas the sparse part corresponds to the abnormal motions of a longitudinal deformation. Experiments have been carried out on both synthetic and real video sequences. Results demonstrate the efficacy of our proposed model to decompose a longitudinal deformation into regular and irregular components.

Straightness metric for measurement of turbulence-induced distortion in long-range imaging.

Algorithms used for mitigation of the effects of atmospheric turbulence on video sequences often rely on a process for creating a reference image to register all of the frames. Because such a pristine image is generally not available, no-reference image quality metrics can be used to identify frames in a sequence that have minimum distortion. Here, we propose a metric that quantifies image warping by measuring image straightness based on line detection. The average length of straight lines in a frame is used to select best frames in a sequence and to generate a reference frame for a subsequent dewarping algorithm. We perform tests with this metric on simulated data that exhibits varying degrees of distortion and blur and spans normalized turbulence strengths between 0.75 and 4.5. We show, through these simulations, that the metric can differentiate between weak and moderate turbulence effects. We also show in simulations that the optical flow that uses a reference frame generated by this metric produces consistently improved image quality. This improvement is even higher when we employ the metric to guide optical flow that is applied to three real video sequences taken over a 7 km path.

Imagery of negative interpersonal experiences influence the neural mechanisms of social interaction

Negative interpersonal experiences are a key contributor to psychiatric disorders. While previous research has shown that negative interpersonal experiences influence social cognition, less is known about the effects on participation in social interactions and the underlying neurobiology. To address this, we developed a new naturalistic version of a gaze-contingent paradigm using real video sequences of gaze behaviour that respond to the participants’ gaze in real-time in order to create a believable and continuous interactive social situation. Additionally, participants listened to two autobiographical audio-scripts that guided them to imagine a recent stressful and a relaxing situation and performed the gaze-based social interaction task before and after the presentation of either the stressful or the relaxing audio-script. Our results demonstrate that the social interaction task robustly recruits brain areas with known involvement in social cognition, namely the medial prefrontal cortex, bilateral temporoparietal junction, superior temporal sulcus as well as the precuneus. Imagery of negative interpersonal experiences compared to relaxing imagery led to a prolonged change in affective state and to increased brain responses during the subsequent social interaction paradigm in the temporoparietal junction, medial prefrontal cortex, anterior cingulate cortex, precuneus and inferior frontal gyrus. Taken together this study presents a new naturalistic social interaction paradigm suitable to study the neural mechanisms of social interaction and the results demonstrate that the imagery of negative interpersonal experiences affects social interaction on neural levels.

Automatic estimation of optimal UAV flight parameters for real-time wide areas monitoring

In the last few years, small-scale Unmanned Aerial Vehicles (UAVs) have been used in many video-based monitoring applications, such as search and rescue (SAR) operations, border control, precision agriculture, and many others. Usually, during these missions, a human operator manually selects UAV flight parameters according to the specific monitoring application to be performed. Anyway, regardless a particular mission, some main tasks can be considered common preprocessing steps that require to be accomplished. These tasks include the mosaicking of areas of interest, the detection of changes over time on these areas, finally, the classification of what is present on the ground. The success of these tasks strictly depends on flight and video sensor parameters. In this paper, for the first time in the literature, a method to automatically estimate the optimal parameters, in particular altitude and frame rate, to accomplish the three main tasks reported above is presented and tested. The parameters are estimated according to several factors, including size of the target to be analysed, cruise speed of UAVs, and main internal parameters of the video sensor, i.e., focal length, field of view, and size of the pixel. The full effectiveness of the proposed method, on the three case studies (i.e., main tasks), was proven both by synthetic videos generated with the Aerial Informatics and Robotics Simulation (AirSim) and by real video sequences reported in the UAV Mosaicking and Change Detection (UMCD) and NPU Drone-Map datasets.

An improved YOLO-based road traffic monitoring system

The growing population in large cities is creating traffic management issues. The metropolis road network management also requires constant monitoring, timely expansion, and modernization. In order to handle road traffic issues, an intelligent traffic management solution is required. Intelligent monitoring of traffic involves the detection and tracking of vehicles on roads and highways. There are various sensors for collecting motion information, such as transport video detectors, microwave radars, infrared sensors, ultrasonic sensors, passive acoustic sensors, and others. In this paper, we present an intelligent video surveillance-based vehicle tracking system. The proposed system uses a combination of the neural network, image-based tracking, and You Only Look Once (YOLOv3) to track vehicles. We train the proposed system with different datasets. Moreover, we use real video sequences of road traffic to test the performance of the proposed system. The evaluation outcomes showed that the proposed system can detect, track, and count the vehicles with acceptable results in changing scenarios.

Online Rain/Snow Removal From Surveillance Videos.

Video rain/snow removal from surveillance videos is an important task in the computer vision community since rain/snow existed in videos can severely degenerate the performance of many surveillance system. Various methods have been investigated extensively, but most only consider consistent rain/snow under stable background scenes. Rain/snow captured from practical surveillance camera, however, is always highly dynamic in time, and those videos also include occasionally transformed background scenes and background motions caused by waving leaves or water surfaces. To this issue, this paper proposes a novel rain/snow removal approach, which fully considers dynamic statistics of both rain/snow and background scenes taken from a video sequence. Specifically, the rain/snow is encoded as an online multi-scale convolutional sparse coding (OMS-CSC) model, which not only finely delivers the sparse scattering and multi-scale shapes of real rain/snow, but also well distinguish the components of background motion from rain/snow layer. The real-time ameliorated parameters in the model well encodes their temporally dynamic configurations. Furthermore, a transformation operator imposed on the background scenes is further embedded into the proposed model, which finely conveys the background transformations, such as rotations, scalings and distortions, inevitably existed in a real video sequence. The approach so constructed can naturally better adapt to the dynamic rain/snow as well as background changes, and also suitable to deal with the streaming video attributed its online learning mode. The proposed model is formulated in a concise maximum a posterior (MAP) framework and is readily solved by the alternating direction method of multipliers (ADMM). Compared with the state-of-the-art online and offline video rain/snow removal methods, the proposed method achieves best performance on synthetic and real videos datasets both visually and quantitatively. Specifically, our method can be implemented in relatively high efficiency, showing its potential to real-time video rain/snow removal. The code page is at: https://github.com/MinghanLi/OTMSCSC_matlab_2020.

VisionBlender: a tool to efficiently generate computer vision datasets for robotic surgery

ABSTRACT Surgical robots rely on robust and efficient computer vision algorithms to be able to intervene in real-time. The main problem, however, is that the training or testing of such algorithms, especially when using deep learning techniques, requires large endoscopic datasets which are challenging to obtain, since they require expensive hardware, ethical approvals, patient consent and access to hospitals. This paper presents VisionBlender, a solution to efficiently generate large and accurate endoscopic datasets for validating surgical vision algorithms. VisionBlender is a synthetic dataset generator that adds a user interface to Blender, allowing users to generate realistic video sequences with ground truth maps of depth, disparity, segmentation masks, surface normals, optical flow, object pose, and camera parameters. VisionBlender was built with special focus on robotic surgery, and examples of endoscopic data that can be generated using this tool are presented. Possible applications are also discussed, and here we present one of those applications where the generated data has been used to train and evaluate state-of-the-art 3D reconstruction algorithms. Being able to generate realistic endoscopic datasets efficiently, VisionBlender promises an exciting step forward in robotic surgery.

NRSfPP: non-rigid structure-from-perspective projection

A state-of-the-art algorithm for perspective projection reconstruction of non-rigid surfaces from single-view and realistic videos is proposed. It overcomes the limitations arising from the usage of orthographic camera model and also the complexity and non-linearity issues of perspective projection equation. Unlike traditional non-rigid structure-from-motion (NRSfM) methods, which have been studied only on synthetic datasets and controlled lab environments that require some prior constraints (such as manually segmented objects, limited rotations and occlusions, and full-length trajectories); the proposed method can be used in realistic video sequences. In addition, contrary to previous methods that use multiple cameras with different relative viewing angles, only a single-view video is required to reconstruct the 3D structures. By only using the 2D frames of incoming video stream, the proposed method extracts the projective depth coefficients of each point in each input frame, rotation matrix, translation vector, varying camera parameters (such as focal lengths for each input frame), and finally reconstructs the 3D deformable shape. Due to the high number of unknowns, the problem has been divided into two parts of projective depth coefficients extraction and 3D shape reconstruction. Perspective reconstruction of non-rigid surfaces has been extended to be certainly converged which leads to a significant increase in execution frequency of the iterated algorithm that has been presented for projective depth coefficients extraction. As such, it produces promising results for perspective projection reconstruction of non-rigid surfaces from single-view and realistic videos. The accuracy and robustness of the proposed method is demonstrated quantitatively on synthetic data and qualitatively on real image sequences. The experimental results show that NRSfPP provides the state-of-the-art results and resolves the failures of previous approaches by eliminating all of the predefined situations and constraints of applying perspective projection as the camera model.

Swimmer’s Head Detection Based on a Contrario and Scaled Composite JTC Approaches

In order to accompany the swimming coaches in evaluating high-level swimmers, we developed a prototype for instantaneous speed estimation. To achieve this, we proposed and validated, in a previous work, a swimmer tracking system based on data fusion. However, the initialization phase is done manually, and our aim, in this paper, is to automate this process. First, we propose a region of interest localization module that allows the detection of the first appearance of the swimmer in the lane as well as the restriction of the region of interest around him. This module is based on the method a contrario which consists of modeling the random noise corresponding to the water and detecting the structured movement relative to the swimmer motion. To do that, we calibrate the pool using DLT (Direct Linear Transform) technique, extract the concerned lane, apply the frame difference approach to detect the moving objects, and then decompose the lane into blocs and classify them into swimmer motion or noise. Second, in order to detect the swimmer’s head, we propose the Scaled Composite JTC which is based on the NL-JTC correlation technique. The input plane of this latter includes a target and a reference image. The first is the region of interest detected by the method a contrario. The second consists of a Scaled Composite Reference. The tests conducted on real video sequences of French swimming championships (Limoges 2015) showed very good results in terms of region of interest localization and swimmer’s head detection which allows a reliable initialization for the tracking system.