Multiple Video Sequences Research Articles

Fully Convolutional Encoder-Decoder With an Attention Mechanism for Practical Pedestrian Trajectory Prediction

Pedestrian trajectory prediction using video is essential for many practical traffic applications. Most existing pedestrian trajectory prediction methods are based on fully connected long short-term memory (LSTM) networks and perform well on public datasets. However, these methods still have three defects: a) Most of them rely on manual annotations to obtain information about the environment surrounding the subject pedestrian, which limits practical applications; b) The interaction among pedestrians and obstacles in a scene is little studied, which leads to greater prediction error; c) Traditional LSTM methods are based on the previous moment and ignore the correlation between the future and distant past states of the pedestrian, which generates unrealistic trajectories. To tackle these problems, first, in the stage of data processing, we use an image semantic segmentation algorithm to obtain multi-category obstacle information and design an end-to-end “Siamese Position Extraction” model to obtain more accurate pedestrian interaction data. Second, we design an end-to-end fully convolutional LSTM encoder-decoder with an attention mechanism (FLEAM) to overcome the shortcomings of LSTM. Third, we compare FLEAM with several state-of-the-art LSTM-based prediction methods on multiple video sequences in the datasets ETH, UCY and MOT20. The results show that our approach generates the same prediction error as the best results of the state-of-the-art method. However, FLEAM has more potential for practice application because it does not rely on manually annotated data. We further validate the effectiveness of FLEAM by employing manually annotated data, finding that it generates much less prediction error.

Review on recent Computer Vision Methods for Human Action Recognition


 
 
 The subject of human activity recognition is considered an important goal in the domain of computer vision from the beginning of its development and has reached new levels. It is also thought of as a simple procedure. Problems arise in fast-moving and advanced scenes, and the numerical analysis of artificial intelligence (AI) through activity prediction mistreatment increased the attention of researchers to study. Having decent methodological and content related variations, several datasets were created to address the evaluation of these ways. Human activities play an important role but with challenging characteristic in various fields. Many applications exist in this field, such as smart home, helpful AI, HCI (Human-Computer Interaction), advancements in protection in applications such as transportation, education, security, and medication management, including falling or helping elderly in medical drug consumption. The positive impact of deep learning techniques on many vision applications leads to deploying these ways in video processing. Analysis of human behavior activities involves major challenges when human presence is concerned. One individual can be represented in multiple video sequences through skeleton, motion and/or abstract characteristics. This work aims to address human presence by combining many options and utilizing a new RNN structure for activities. The paper focuses on recent advances in machine learning-assisted action recognition.
 Existing modern techniques for the recognition of actions and prediction similarly because the future scope for the analysis is mentioned accuracy within the review paper.
 
 

Quadruple tripatch-wise modular architecture-based real-time structure from motion

Structure from motion (SFM), a research hotspot in the intelligent transportation field (including autonomous driving, environmental perception and augmented reality (AR) for artificial intelligence terminals), can automatically recover ego-motion state estimations and 3D scene reconstructions from multiple images or video sequences. Most existing vision methods operate offline in indoor scenes, and their reconstruction accuracies greatly depend on the tracking lifetimes and accuracies of feature points. Reprojection matrix and redundant regression noise computations are exponential disasters for the calculation and reconstruction drift of large-scale scenes. This paper proposes a quadruple tripatch-wise modular architecture (QTMA) for autonomous vehicle stereo image sequences that decomposes rigid scenes into nonrigid motion-segmented pieces for reconstruction. An advanced energy function for salient image features is established by combining multiple feature types with weighted finite-element mesh. Closed quadruple annular matching and relocation are performed via multiresolution pyramid images. The proposed incremental integral-mapping calculation method and unique tree-like stacked storage containers prolong the tracking lifetimes of consecutive frames and ensure the spatiotemporal consistency and robustness of the homonymous image features in different subsequences. Experimental results verify the effectiveness of this architecture for different transportation scenes; the frame rate processing speed reaches 30 fps, the calculation accuracy regarding the path distance difference reaches 99.49%, and the estimation results regarding the maximum speeds of motion are closer to the ground truth. The translation error of the motion pose is 0.0136%, and the rotation error is 0.0035 [deg/m], which has more yaw stability than the existing state-of-the-art methods. Furthermore, in the reconstructed point cloud quality demonstration, the mean value of roughness is reduced by 40.127%, the mean value of density is improved by 27.701%, and the accuracy reaches 91.149% within a certain distance tolerance. This paper has significant theoretical research value and application potential for positioning, path tracking, and navigation in adaptive cruise control (ACC) and advanced driver assistance systems (ADAS).

Camera Selection for Occlusion-Less Surgery Recording via Training With an Egocentric Camera

Recording surgery is an important technique for education and the evaluation of medical treatments. However, capturing targets such as the surgical field, surgical tools, and the surgeon’s hands, is almost impossible since these targets are heavily occluded by the surgeon’s head and body during a surgery. We used a recording system in which multiple cameras are installed in the surgical lump, supposing at least one camera would capture the target without occlusion. As this system records multiple video sequences, we address the task to select a best view camera automatically. Recently, learning-based approaches in a fully supervised manner have been proposed for this task, but these previous approaches completely rely on manual annotation of the training data. In this paper, we focus on the eye tracker mounted on the surgeon’s head, which can capture the recording targets without occlusion. Employing this first-person-view video synchronized with multiple videos of the surgical lump, we propose a novel camera selection approach using a self-supervised learning framework. In experiments, we created a dataset composed of four different breast surgery. Our extended experiments showed that our approach successfully switched to the best camera view without manual annotation and achieved competitive accuracy compared with conventional supervised methods. Also, our approach yielded effective visual representations comparable to state-of-the-art self-supervised learning frameworks.

Hybrid Ultra-Wide Viewing Broadcasting Platform With Reduced Keypoint Redundancy Matching and Robust Blending for Parallax Errors

Advanced Television Systems Committee (ATSC) 3.0 is a more innovative standardization including Ultra High Definition (UHD) broadcasting and mobile High-Definition (HD) broadcasting. It also includes the deployment of a disaster information alarm function as well as the adoption of All-Internet Protocol (IP) based transmission in terrestrial broadcasting networks. Thus, ATSC enables additional technologies such as hybrid broadcasting services by interworking terrestrial broadcasting and internet networks for bidirectional customized services. In addition, there is a growing interest in wider viewing contents, which provide more immersive contents than existing UHD broadcasting mediums. Wider viewing contents can be acquired by using large field of view (FoV) lenses such as fish-eye lenses or by stitching several sub-images. The images captured by large FoV lenses have heavily curved surface distortions, while the distortion of the stitching-based wider viewing contents caused by the synthesizing process on overlapping regions is lower. Provision of wider viewing contents over the traditional broadcasting service has limitations in terms of bandwidth. However, they can be overcome by using hybrid broadcasting platforms such as ATSC 3.0, which transmits a video sequence over an RF channel and other viewing-angle video sequences through the Internet network, where signalling among those video sequences is required. Therefore, this paper proposes a hybrid broadcasting platform for transmitting multiple video sequences over broadcasting and Internet networks, and stitching them for wider viewing.

Vision-Based Monitoring of Flare Soot

The flare stack is a typical flare gas combustion facility used to guarantee the safe production of petrochemical plants, refineries, and other enterprises. One of the most vital problems of a flare stack is the incomplete combustion of flare gas, which produces a large amount of flare soot and, thus, endangers the atmosphere and human health. Hence, an effective and efficient flare soot monitoring system that has important guiding significance to environmental protection is strongly required. To this end, we devise a vision-based monitor of flare soot (VMFS) that can search for flare soot in a timely way and ensure the full combustion of flare gas. First, the proposed VMFS leverages the broadly tuned color channel to recognize a flame in an input video frame since the flame is the source of flare soot in our application. Second, our monitor incorporates fast saliency detection with K-means to fix the position of the flame. Third, we take the flame area as the center to search for the potential flare soot region, followed by identifying the flare soot based on the background color channel. The results of experiments on multiple video sequences collected at a real petrochemical plant reveal that the proposed VMFS is superior to state-of-the-art relevant models in both monitoring performance and computational efficiency. The implementation code will soon be released at https://kegu.netlify.com/.

A Pseudo-likelihood Approach for Geo-localization of Events from Crowd-sourced Sensor-Metadata

Events such as live concerts, protest marches, and exhibitions are often video recorded by many people at the same time, typically using smartphone devices. In this work, we address the problem of geo-localizing such events from crowd-generated data. Traditional approaches for solving such a problem using multiple video sequences of the event would require highly complex computer vision (CV) methods, which are computation intensive and are not robust under the environment where visual data are collected through crowd-sourced medium. In the present work, we approach the problem in a probabilistic framework using only the sensor metadata obtained from smartphones. We model the event location and camera locations and orientations (camera parameters) as the hidden states in a Hidden Markov Model. The sensor metadata from GPS and the digital compass from user smartphones are used as the observations associated with the hidden states of the model. We have used a suitable potential function to capture the complex interaction between the hidden states (i.e., event location and camera parameters). The non-Gaussian densities involved in the model, such as the potential function involving hidden states, make the maximum-likelihood estimation intractable. We propose a pseudo-likelihood-based approach to maximize the approximate-likelihood, which provides a tractable solution to the problem. The experimental results on the simulated as well as real data show correct event geo-localization using the proposed method. When compared with several baselines the proposed method shows a superior performance. The overall computation time required is much smaller, since only the sensor metadata are used instead of visual data.

A scale transform and deformation target tracking algorithm based on correlation filtering

In order to improve the robustness of the correlation filtering (CF) tracking algorithm and overcome the problem that the traditional correlation filtering method can not deal with the target’s scale and deformation change, a feature-adaptive scale adaptive correlation filter tracking algorithm is proposed. Firstly, the target is characterized; then the output filter is calculated by using the correlation filter; finally, the image blocks of different scales are intercepted from the target position of the current frame, and the optimal estimation of the target scale is obtained by the adaptive scale pool model. The experiment selects multiple video sequences for testing and compares the proposed algorithm with other target tracking methods. The experimental results show that the average performance is better than the comparison method.

A Convex Optimization Framework for Video Quality and Resolution Enhancement From Multiple Descriptions.

Transmission and compression technologies advancement over the past decade led to a shift of multimedia content towards cloud systems. Multiple copies of the same video are available through numerous distribution systems. Different compression levels, algorithms and resolutions are used to match the requirements of particular applications. As 4k display technologies are rapidly adopted, resolution enhancement algorithms are of vital importance. Current solutions do not take into account the particularities of different video encoders, while video reconstruction methods from compressed sources do not provide resolution enhancement. In this paper, we propose a multi source compressed video enhancement framework, where each description can have a different compression level and resolution. Using a variational formulation based on a modern proximal dual splitting algorithm, we efficiently combine multiple descriptions of the same video. Two applications are proposed: combining two compressed low resolution (LR) descriptions of a video sequence into a high resolution (HR) description and enhancing a compressed HR video using a LR compressed description. Tests are performed over multiple video sequences encoded with high efficiency video coding, at different compression levels and resolutions obtained through multiple down-sampling methods.

Background Subtraction Using Spatio-Temporal Group Sparsity Recovery

Background subtraction is a key step in a wide spectrum of video applications, such as object tracking and human behavior analysis. Compressive sensing-based methods, which make little specific assumptions about the background, have recently attracted wide attention in background subtraction. Within the framework of compressive sensing, background subtraction is solved as a decomposition and optimization problem, where the foreground is typically modeled as pixel-wised sparse outliers. However, in real videos, foreground pixels are often not randomly distributed, but instead, group clustered. Moreover, due to costly computational expenses, most compressive sensing-based methods are unable to process frames online. In this paper, we take into account the group properties of foreground signals in both spatial and temporal domains, and propose a greedy pursuit-based method called spatio-temporal group sparsity recovery, which prunes data residues in an iterative process, according to both sparsity and group clustering priors, rather than merely sparsity. Furthermore, a random strategy for background dictionary learning is used to handle complex background variations, while foreground-free training is not required. Finally, we propose a two-pass framework to achieve online processing. The proposed method is validated on multiple challenging video sequences. Experiments demonstrate that our approach effectively works on a wide range of complex scenarios and achieves a state-of-the-art performance with far fewer computations.

Spatio-temporal super-resolution for multi-videos based on belief propagation

Aiming at improving spatio-temporal resolution of video for real-world applications, this paper focuses on the spatio-temporal super-resolution reconstruction algorithm for multiple asynchronous video sequences. The current multi-video super-resolution algorithms still have some weaknesses in application such as oversimplification of motion, unknown blurring and noise level. For further improvement, a Maximum Posterior Likelihood-Markov Random Field (MAP-MRF) based super-resolution reconstruction method is proposed and proved robust in achieving the real-world super-resolution imaging. The proposed method adopts weighted 3D neighborhood system (WNS) MAP-MRF model to accurately describe the spatial and temporal correlations between multiple video sequences. In order to improve the estimation accuracy of the motions for complex scenes, the improved Scale-Invariant Feature Transform (SIFT) Flow algorithm based on sparsity in wavelet domain is proposed, which can afford the large displacement, rotational movement and other complexities in asynchronous multi-video sequences. The Belief Propagation (BP) algorithm is applied to estimate the parameters of MAP-MRF model, such as motion vector and the super-resolution images in an iterative coarse-to-fine scheme. With the proposed algorithms mentioned above, MAP-MRF based super-resolution reconstruction method has better capabilities of edge sharpness and detailed texture preserving, and robustness of noise suppressing. The experimental result has confirmed the effectiveness of the proposed method under the practical conditions.

Fast Smoke Detection for Video Surveillance Using CUDA

Smoke detection is a key component of disaster and accident detection. Despite the wide variety of smoke detection methods and sensors that have been proposed, none has been able to maintain a high frame rate while improving detection performance. In this paper, a smoke detection method for surveillance cameras is presented that relies on shape features of smoke regions as well as color information. The method takes advantage of the use of a stationary camera by using a background subtraction method to detect changes in the scene. The color of the smoke is used to assess the probability that pixels in the scene belong to a smoke region. Due to the variable density of the smoke, not all pixels of the actual smoke area appear in the foreground mask. These separate pixels are united by morphological operations and connected-component labeling methods. The existence of a smoke region is confirmed by analyzing the roughness of its boundary. The final step of the algorithm is to check the density of edge pixels within a region. Comparison of objects in the current and previous frames is conducted to distinguish fluid smoke regions from rigid moving objects. Some parts of the algorithm were boosted by means of parallel processing using compute unified device architecture graphics processing unit, thereby enabling fast processing of both low-resolution and high-definition videos. The algorithm was tested on multiple video sequences and demonstrated appropriate processing time for a realistic range of frame sizes.

Efficient rate control scheme using modified inter-layer dependency for spatial scalability

Scalable video coding extension has been added to H.264AVC to support compression and encoding of multiple resolution video sequences, having different frame rates and fidelities in a single bit stream. The motion vectors and the residual data of the enhancement layers are derived from up-sampling the co-located macroblock (MB) of the base layer. The peak signal to noise ratio (PSNR) across the enhancement layers is degraded as up-sampling introduces distortion of high-frequency components. In this paper, a spatial-resolution-ratio-based MB mode decision scheme is proposed for spatially enhanced layers. The scheme uses the motion estimated at the base layer, to encode the respective MBs in the enhancement layers. The spatial–temporal search schemes at the enhancement layers are used to derive motion vectors and residues that are encoded using a quantization parameter obtained using independent rate control (IRC) scheme. The IRC from the prior art is modified to achieve better rate control per layer by recursive updates for mean absolute difference values of each basic unit. Proposed modified inter-layer dependency shows improvement in the PSNR for enhancement layers while the updated IRC enforces better IRC for all the layers.

Parallel Multiview Video Coding Exploiting Group of Pictures Level Parallelism

This paper presents the use of a computer cluster with heterogeneous computing components to provide concurrency and multi-level parallelism at coarse grain and massive fine-grain for multiview video coding (MVC) applications. MVC involves coding of multiple video sequences that are taken from the same scene but different perspective. In addition to motion estimation (ME) used in conventional video coding for single view video for exploiting inter-frame temporal similarities, MVC adopts disparity estimation (DE) to further increase compression. To overcome the huge computational cost associated with ME and by extension with DE, attention has been mainly focused on developing fast ME/DE algorithms. Although fast ME/DE algorithms bring substantial speedup, to achieve realtime MVC encoding, it requires further acceleration of the coding process at higher levels. Towards this end, this paper proposes a multiple-view-parallel, multiple-interleaved group of pictures (multiple-IGOP) scheduling scheme for MVC. When evaluated over eight views, with no loss in rate distortion (RD) performance, the proposed scheme outperforms view-sequential coding by a factor of up to 12.4 and 12.3, respectively, for two popular prediction structures, IBP and IPP.

Optimal reference view selection algorithm for low complexity disparity estimation

Multi-view video coding (MVC) is composed of multiple video sequences captured simultaneously by multiple closely spaced cameras. Such a system demands a higher computational complexity due to the increased number of cameras. Disparity estimation (DE) is one of the compression techniques used to remove inter-view redundancy in MVC, which causes high complexity. In order to overcome the problem of high computational complexity in real-time MVC, an optimal reference view selection algorithm is proposed. The proposed scheme decreases the coding complexity by reducing one-directional DE operation and minimizing the decrease in coding performance in MVC.

A DASH-based HEVC multi-view video streaming system

Recent advancement in cameras and image processing technology has generated a paradigm shift from traditional 2D and 3D video to multi-view video (MVV) technology, while at the same time improving video quality and compression through standards such as high efficiency video coding (HEVC). In multi-view, cameras are placed in predetermined positions to capture the video from various views. Delivering such views with high quality over the Internet is a challenging prospect, as MVV traffic is several times larger than traditional video, since it consists of multiple video sequences, each captured from a different angle, requiring more bandwidth than single-view video to transmit MVV. In addition, the Internet is known to be prone to packet loss, delay, and bandwidth variation, which adversely affect MVV transmission. Another challenge is that end users' devices have different capabilities in terms of computing power, display, and access link capacity, requiring MVV to be adapted to each user's context. In this paper, we propose an HEVC multi-view system using Dynamic Adaptive Streaming over HTTP to overcome the above-mentioned challenges. Our system uses an adaptive mechanism to adjust the video bit rate to the variations of bandwidth in best effort networks. We also propose a novel scalable way for the multi-view video and depth content for 3D video in terms of the number of transmitted views. Our objective measurements show that our method of transmitting MVV content can maximize the perceptual quality of virtual views after the rendering and hence increase the user's quality of experience.

Tracking the joints of arthropod legs using multiple images and inverse kinematics

Negligible research has been conducted on the systematic analysis of small animals, such as lizards and arthropods. Furthermore, few studies in the area use digitized data obtained from animals. Currently, the motion analysis of most living creatures is carried out using optical motion capture systems. However, these systems are inapplicable to small animals because of the difficulty in attaching optical markers to them. Ongoing research on markerless motion capture technology aims to solve this problem, but most of those studies have been conducted on human subjects. Therefore, more studies are needed to apply a markerless motion capture technology to insects with several legs and joints. In this paper, we propose a method to calculate the continuous pose of small articulated animals using multiple view video sequences. This includes the extraction of the initial skeleton model and the extraction of roots and extremities, as well as the calculation of joint kinematics using the Forward and Backward Reaching Inverse Kinematics (FABRIK) algorithm. We also generate the optimal three-dimensional skeleton model using the correction method in order to obtain the position of the joints.

Compressive Background Modeling for Foreground Extraction

Robust and efficient foreground extraction is a crucial topic in many computer vision applications. In this paper, we propose an accurate and computationally efficient background subtraction method. The key idea is to reduce the data dimensionality of image frame based on compressive sensing and in the meanwhile apply sparse representation to build the current background by a set of preceding background images. According to greedy iterative optimization, the background image and background subtracted image can be recovered by using a few compressive measurements. The proposed method is validated through multiple challenging video sequences. Experimental results demonstrate the fact that the performance of our approach is comparable to those of existing classical background subtraction techniques.

The Optimal Partition Algorithm for Multiple Video Sequences in Streaming Media System

The fine grained scalable (FGS) coding is one of the most popular coding technologies in network video stream. In FGS stream, the base layer rate is an important parameter and should generally make a compromise between the number of beneficial client and the efficiency of video coding. To maximize all clients' perceived quality, the optimal partition algorithm for multiple FGS-coded video is proposed. By analyzing the algorithms oriented FGS-coded stream proposed by Hsu et al, this paper first puts forward a method to build the R-Q function curve and proposes an algorithm to cut search space of finding the optimal base layer rate. Second, according to the optimization problem of multiple sequences, the improved greedy algorithm MFGS_N are proposed to improve the probability to find the optimal base layer rate, thus to reduce the runtime. Moreover, illustration examples are given to demonstrate and compare the effect of the algorithm.

Pedestrian Evacuation Time Model for Urban Metro Hubs Based on Multiple Video Sequences Data

Evacuation time is a significant safety coefficient for Urban Metro Hubs (UMHs). Usually, a reasonable model for evacuation time will effectively promote the safety for pedestrian when emergency incidents occur in UMHs. In this paper, we propose a pedestrian evacuation time model for UMHs to improve the accuracy and reliability of its evacuation time. Firstly, we design an experiment survey based on the multiple video sequences to analyze the characteristics of pedestrian flow. Then, we decompose the evacuation process on the basis of the parameters, which involve the evacuation characteristics, the speed-density variation law, the pedestrian drop-off time, the platform evacuation time, and the channel evacuation time. Finally, we take the Bei Da-jie metro hub in Xi’an as an example, and verify the feasibility of the proposed pedestrian evacuation time model. The results show that the relative error for the evacuation time between the experiment result and the actual data is only 1.90%, where the experiment time is 169.87 s and the actual time is 166.64 s. Moreover, the proposed model strictly follows the Code for Design of Metro (GB 50157-003) and hence it can provide a good theoretical guidance for innovating the evacuation efficiency and the design reasonability of UMHs.