Real-world Video Research Articles

Ordered smooth representation clustering

The smooth representation (SMR) model is a widely used segmentation method in computer vision. This model adopts the K nearest neighbour (KNN) graph to select samples for representation. All neighbours in the KNN graph are assumed to be equally important candidates. In this paper, we use special weights that are calculated by a novel cross-view kernel function to evaluate the contributions of neighbours to the subspace clustering in SMR. The neighbours that are found by the Gaussian similarity formula can be considered long-range similar neighbours. We add another item to accurately reflect the order relation in the cross-view kernel function. This addition allows the kernel function to generalize the conventional SMR method for sequential data. The ordered smooth representation (OSMR) model outperforms other representative space clustering methods on public datasets, namely, the UCI database, the USPS database, Yale B datasets, the Freiburg–Berkeley Motion Segmentation database and a real-world mobile video that was captured by a smart phone.

Traffic extreme situations detection in video sequences based on integral optical flow

Road traffic analysis is an important task in many applications and it can be used in video surveillance systems to prevent many undesirable events. In this paper, we propose a new method based on integral optical flow to analyze cars movement in video and detect flow extreme situations in real-world videos. Firstly, integral optical flow is calculated for video sequences based on optical flow, thus random background motion is eliminated; secondly, pixel-level motion maps which describe cars movement from different perspectives are created based on integral optical flow; thirdly, region-level indicators are defined and calculated; finally, threshold segmentation is used to identify different cars movements. We also define and calculate several parameters of moving car flow including direction, speed, density, and intensity without detecting and counting cars. Experimental results show that our method can identify cars directional movement, cars divergence and cars accumulation effectively.

Deep video-to-video transformations for accessibility with an application to photosensitivity

• New approach to accessibility, deep video-to-video transformations. • An application to photosensitivity where networks learn flash suppression. • An oracle and a GAN make problematic videos and then remove those issues. • Scales to huge datasets and works on real videos that triggered seizures. • Generic non-symbolic approach applicable to many other accessibility issues. We demonstrate how to construct a new class of visual assistive technologies that, rather than extract symbolic information, learn to transform the visual environment to make it more accessible. We do so without engineering which transformations are useful allowing for arbitrary modifications of the visual input. As an instantiation of this idea we tackle a problem that affects and hurts millions worldwide: photosensitivity. Any time an affected person opens a website, video, or some other medium that contains an adverse visual stimulus, either intended or unintended, they might experience a seizure with potentially significant consequences. We show how a deep network can learn a video-to-video transformation rendering such stimuli harmless while otherwise preserving the video. This approach uses a specification of the adverse phenomena, the forward transformation, to learn the inverse transformation. We show how such a network generalizes to real-world videos that have triggered numerous seizures, both by mistake and in politically-motivated attacks. A number of complimentary approaches are demonstrated including using a hand-crafted generator and a GAN using a differentiable perceptual metric. Such technology can be deployed offline to protect videos before they are shown or online with assistive glasses or real-time post processing. Other applications of this general technique include helping those with limited vision, attention deficit hyperactivity disorder, and autism.

Object Detection During Newborn Resuscitation Activities.

Birth asphyxia is a major newborn mortality problem in low-resource countries. International guideline provides treatment recommendations; however, the importance and effect of the different treatments are not fully explored. The available data are collected in Tanzania, during newborn resuscitation, for analysis of the resuscitation activities and the response of the newborn. An important step in the analysis is to create activity timelines of the episodes, where activities include ventilation, suction, stimulation, etc. Methods: The available recordings are noisy real-world videos with large variations. We propose a two-step process in order to detect activities possibly overlapping in time. The first step is to detect and track the relevant objects, such as bag-mask resuscitator, heart rate sensors, etc., and the second step is to use this information to recognize the resuscitation activities. The topic of this paper is the first step, and the object detection and tracking are based on convolutional neural networks followed by post processing. The performance of the object detection during activities were 96.97% (ventilations), 100% (attaching/removing heart rate sensor), and 75% (suction) on a test set of 20 videos. The system also estimate the number of health care providers present with a performance of 71.16%. The proposed object detection and tracking system provides promising results in noisy newborn resuscitation videos. This is the first step in a thorough analysis of newborn resuscitation episodes, which could provide important insight about the importance and effect of different newborn resuscitation activities.

Potential benefits of controlled vehicle braking to reduce pedestrian ground contact injuries

Protecting struck pedestrians during the ground contact phase has been a challenge for decades. Recent studies have shown how ground related injury is influenced by pedestrian kinematics. In this paper we further developed this approach by assessing the potential of controlling vehicle braking to reduce pedestrian ground contact injuries. Applying a recently proposed Simulation Test Sample, a series of simulations were run using the MADYMO software environment. The approach considered 6 vehicle shapes, 4 pedestrian models, 3 impact velocities and 2 pedestrian gaits and each case was considered with two different vehicle braking approaches. The first was full braking, while the second applied controlled braking, for which a strategy based on pedestrian kinematics was applied. The effect of vehicle braking was evaluated using the Weighted Injury Cost (WIC) of overall pedestrian injuries and the pedestrian-ground impact velocity change. The proximity of the vehicle and pedestrian at the instant of ground contact was also evaluated to assess the potential of future vehicle based intervention methods to cushion the ground contact. Finally real-world videos of pedestrian collisions were analyzed to estimate the available free vehicle stopping distances. Results showed substantial median reductions in WIC and head impact velocity for all vehicle shapes except the Van. The proximity of the pedestrian to the vehicle front at the instant of ground contact under controlled braking is less than 1.5 m in most cases, and the required stopping distance for the vehicle under controlled braking was within the available stopping distance estimated from the video footage in about 74% of cases. It is concluded that controlled braking has significant potential to reduce the overall burden of pedestrian ground contact injuries, but future efforts are required to establish an optimized braking strategy as well as a means to handle those cases where controlled braking is not beneficial or even harmful.

Harmonic rendering for visual coherence on mobile outdoor AR environment

Rapid developments in augmented reality (AR) and related technologies have led to increasing interest in immersive content. AR environments are created by combining virtual 3D models with a real-world video background. It is important to merge these two worlds seamlessly if users are to enjoy AR applications, but, all too often, the illumination and shading of virtual objects is not consider the real world lighting condition or does not match that of nearby real objects. In addition, visual artifacts produced when blending real and virtual objects further limit realism. In this paper, we propose a harmonic rendering technique that minimizes the visual discrepancy between the real and virtual environments to maintain visual coherence in outdoor AR. To do this, we introduce a method of estimating and approximating the Sun’s position and the sunlight direction to estimate the real sunlight intensity, as this is the most significant illumination source in outdoor AR and it provides a more realistic lighting environment for such content, reducing the mismatch between real and virtual objects.

Multi-target tracking by non-linear motion patterns based on hierarchical network flows

We propose a multi-target tracking method for pedestrians, through locating them and maintaining their identities to make their individual trajectories. In most existing data association-based tracking approaches, the motion information in the affinity model is always linear, but the motion of target is non-linear in real-world surveillance video, especially when there is occlusion. This paper proposes a global optimization method based on hierarchical network flows with a non-linear motion model to track multiple targets. In our method, each node in network represents a tracklet, and each edge represents the likelihood of neighboring tracklets belonging to the same trajectory measured by affinity score that we proposed, it differs from other network flow formulation whose nodes represent detection responses. A non-linear motion map is used for explaining non-linear motion pattern between neighboring tracklets and getting motion affinity when there are time gaps. We evaluate our approach on 2DMOT2015, MOT16 database, and several real surveillance videos. Experimental results demonstrate that the proposed method can achieve continuous tracking trajectories under the case of motion direction changes and complete occlusions.

A Quality-Retaining Power-Saving Framework for Video Applications on Mobile OLED Displays

As people increase their dependency on mobile applications and services, saving power consumption of mobile devices becomes an important challenge for supporting video streaming applications. This paper investigates how to minimize the power consumption of an organic light-emitting diode (OLED) display when displaying a video under consideration for the user’s visual experience. We first model the minimization problem as an OLED video scaling optimization problem. We then propose an algorithm to solve the optimization problem and prove that the algorithm is optimal in terms of power savings. We next introduce an energy-saving cloud service based on the algorithm. Finally, the results of the extensive experiments conducted on a commercial mobile smartphone with four real-world videos to evaluate the performance of the proposed algorithm are very encouraging.

A multi-image Joint Re-ranking framework with updateable Image Pool for person re-identification

Real-world video surveillance has increasing demand for person re-identification. Existing multi-shot works usually aggregate single sample features by computing the average features or using time series model. The Multi-image Joint Re-ranking framework with updateable Image Pool that we are proposing will give a different approach. First, we defined the term ‘Image Pool’ to store image samples for each pedestrian. Next, the updating rules of Image Pool has been defined in order to optimize the representativeness of it. Second, we compute initial ranking lists of every sample in Image Pool, and propose the ‘Multiple-image Joint Re-ranking’ algorithm to aggregate initial ranking lists. We calculate the rank score of partial elements of initial ranking lists. In the end, we get final ranking list by ascending the order of the rank scores. We validated our re-ranking results on Market-1501, iLIDS-VID, PRID-2011 and our ITSD datasets, and the results outperform other methods.

Dense 3D-Convolutional Neural Network for Person Re-Identification in Videos

Person re-identification aims at identifying a certain pedestrian across non-overlapping multi-camera networks in different time and places. Existing person re-identification approaches mainly focus on matching pedestrians on images; however, little attention has been paid to re-identify pedestrians in videos. Compared to images, video clips contain motion patterns of pedestrians, which is crucial to person re-identification. Moreover, consecutive video frames present pedestrian appearance with different body poses and from different viewpoints, providing valuable information toward addressing the challenge of pose variation, occlusion, and viewpoint change, and so on. In this article, we propose a Dense 3D-Convolutional Network (D3DNet) to jointly learn spatio-temporal and appearance representation for person re-identification in videos. The D3DNet consists of multiple three-dimensional (3D) dense blocks and transition layers. The 3D dense blocks enlarge the receptive fields of visual neurons in both spatial and temporal dimensions, leading to discriminative appearance representation as well as short-term and long-term motion patterns of pedestrians without the requirement of an additional motion estimation module. Moreover, we formulate a loss function consisting of an identification loss and a center loss to minimize intra-class variance and maximize inter-class variance simultaneously, toward addressing the challenge of large intra-class variance and small inter-class variance. Extensive experiments on two real-world video datasets of person identification, i.e., MARS and iLIDS-VID, have shown the effectiveness of the proposed approach.

Motion Maps and Their Applications for Dynamic Object Monitoring

In this paper, dynamic object behavior analysis task is considered. We use basic optical flow to form integral optical flow and use it to separate background and foreground and obtain intensive motion regions. Based on information extracted from integral optical flow, we introduce notion of motion maps and show how it can be used for dynamic object motion analysis. In motion maps, we analyze pixel motions statistically for each frame to obtain quantity of pixels moving toward or away from each position and their comprehensive motion at each position. We then define and compute regional motion indicators to describe motions at regional level. These indicators are further used for analysis of dynamic object behavior. We conduct a set of experiments on real world videos. Experimental results show that motion maps can be effectively used for dynamic object monitoring.

Diversified Crowd Evacuation Method in Large Public Places

Realistic crowd simulation is an important issue for crowd management, public space design, and architectural and urban planning. In this paper, we present a novel approach for planning and directing heterogeneous large-scale crowd in public places. We firstly extract group features from real-world video, and transfer them to the virtual crowd members. On this basis, we perform the crowd evacuation by combining the global path planning and collide avoidance strategy. Our method efficiently produces the most promising paths in both time and space and offers a reasonable distribution of the crowd' members over these paths such that their average evacuation time is minimized based on the probabilistic roadmaps approach. The generated paths are combined with reciprocal velocity obstacle method to handle collisions and generate the final motions of the individuals. We demonstrate its potential on large public places with 10000 individuals and evaluate the results from our simulations using some quantitative quality metrics. In practice, our method can interactively simulate large crowds with thousands of individuals on a desktop PC and naturally generates a diverse set of emergent behaviors.

Activity Interaction Detection by Using Causal Discovery With Order Estimation

Interaction detection is a fundamental task in video activity analysis. Activities contain group structure and temporal order information, which makes interactions complex. In this paper, a novel framework is proposed to explore the global and local activity interactions by using Granger causality discovery in multivariate time series. Based on the inherent properties of time series dependent structures related to variable time orders, an order selection algorithm considering group information is proposed. Experiments on the real world video surveillance dataset show that the activity network constructed by the proposed method is hierarchical, including global and local dependence structure.

Quality-Enhanced OLED Power Savings on Mobile Devices

In the future, mobile systems will increasingly feature more advanced organic light-emitting diode (OLED) displays. The power consumption of these displays is highly dependent on the image content. However, existing OLED power-saving techniques either change the visual experience of users or degrade the visual quality of images in exchange for a reduction in the power consumption. Some techniques attempt to enhance the image quality by employing a compound objective function. In this article, we present a win-win scheme that always enhances the image quality while simultaneously reducing the power consumption. We define metrics to assess the benefits and cost for potential image enhancement and power reduction. We then introduce algorithms that ensure the transformation of images into their quality-enhanced power-saving versions. Next, the win-win scheme is extended to process videos at a justifiable computational cost. All the proposed algorithms are shown to possess the win-win property without assuming accurate OLED power models. Finally, the proposed scheme is realized through a practical camera application and a video camcorder on mobile devices. The results of experiments conducted on a commercial tablet with a popular image database and on a smartphone with real-world videos are very encouraging and provide valuable insights for future research and practices.

Tiny and Dim Infrared Target Detection Based on Weighted Local Contrast

Robust detection of infrared (IR) tiny and dim targets in a single frame remains a hot and difficult problem in military fields. In this letter, we introduce a method for IR tiny and dim target detection based on a new weighted local contrast measure. Our method simultaneously exploits the local contrast of target, the consistency of image background, and the imaging characteristics of the background edges. The proposed method is simple to implement and computationally efficient. We compared our algorithm with six state-of-the-art methods on four real-world videos with different targets and backgrounds. Our method outperforms all the compared algorithms on the ground-truth evaluation with both higher detection rate and lower false alarm rate.

Rain Removal in Traffic Surveillance: Does it Matter?

Varying weather conditions, including rainfall and snowfall, are generally regarded as a challenge for computer vision algorithms. One proposed solution to the challenges induced by rain and snowfall is to artificially remove the rain from images or video using rain removal algorithms. It is the promise of these algorithms that the rain-removed image frames will improve the performance of subsequent segmentation and tracking algorithms. However, rain removal algorithms are typically evaluated on their ability to remove synthetic rain on a small subset of images. Currently, their behavior is unknown on real-world videos when integrated with a typical computer vision pipeline. In this paper, we review the existing rain removal algorithms and propose a new dataset that consists of 22 traffic surveillance sequences under a broad variety of weather conditions that all include either rain or snowfall. We propose a new evaluation protocol that evaluates the rain removal algorithms on their ability to improve the performance of subsequent segmentation, instance segmentation, and feature tracking algorithms under rain and snow. If successful, the de-rained frames of a rain removal algorithm should improve segmentation performance and increase the number of accurately tracked features. The results show that a recent single-frame-based rain removal algorithm increases the segmentation performance by 19.7% on our proposed dataset, but it eventually decreases the feature tracking performance and showed mixed results with recent instance segmentation methods. However, the best video-based rain removal algorithm improves the feature tracking accuracy by 7.72%.

Deep Video Dehazing with Semantic Segmentation.

Recent research have shown the potential of using convolutional neural networks (CNNs) to accomplish single image dehazing. In this work, we take one step further to explore the possibility of exploiting a network to perform haze removal for videos. Unlike single image dehazing, video based approaches can take advantage of the abundant information that exists across neighboring frames. In this work, assuming that a scene point yields highly correlated transmission values between adjacent video frames, we develop a deep learning solution for video dehazing, where a CNN is trained end-to-end to learn how to accumulate information across frames for transmission estimation. The estimated transmission map is subsequently used to recover a haze-free frame via atmospheric scattering model. In addition, as the semantic information of a scene provides a strong prior for image restoration, we propose to incorporate global semantic priors as input to regularize the transmission maps so that the estimated maps can be smooth in the regions of the same object and only discontinuous across the boundaries of different objects. To train this network, we generate a dataset consisted of synthetic hazy and haze-free videos for supervision based on the NYU depth dataset. We show that the features learned from this dataset are capable of removing haze that arises in outdoor scenes in a wide range of videos. Extensive experiments demonstrate that the proposed algorithm performs favorably against the state-of-the-art methods on both synthetic and real-world videos.

Mixture Statistic Metric Learning for Robust Human Action and Expression Recognition

Background objects and textures in real-world video sequences often pose great challenges for human action and facial expression recognition. This paper proposes a mixture statistic metric learning for recognizing human actions and facial expressions in realistic “in the wild” scenarios. In the proposed method, multiple statistics, including temporal means and covariance matrices, as well as parameters of spatial Gaussian mixture distributions, are explicitly mapped to or generated on symmetric positive definite Riemannian manifolds. An implicit mixture of Mahalanobis metrics is learned from the Riemannian manifolds. The learned metrics place similar pairs in local neighborhoods and dissimilar pairs in relatively orthogonal regions on a regularized manifold. The proposed metric learning method also explores the prior distributions within the multiple statistics in the video sequences. The proposed method is tested on five action video data sets and three facial expression data sets and is compared with various state-of-the-art methods. Recognition accuracy and computational efficiency are evaluated in terms of average recognition rates and computational times in seconds, respectively. Competitive performances achieved on both action and facial expression recognition tasks demonstrate the effectiveness of the proposed method.

SeaShips: A Large-Scale Precisely Annotated Dataset for Ship Detection

In this paper, we introduce a new large-scale dataset of ships, called SeaShips, which is designed for training and evaluating ship object detection algorithms. The dataset currently consists of 31 455 images and covers six common ship types (ore carrier, bulk cargo carrier, general cargo ship, container ship, fishing boat, and passenger ship). All of the images are from about 10 080 real-world video segments, which are acquired by the monitoring cameras in a deployed coastline video surveillance system. They are carefully selected to mostly cover all possible imaging variations, for example, different scales, hull parts, illumination, viewpoints, backgrounds, and occlusions. All images are annotated with ship-type labels and high-precision bounding boxes. Based on the SeaShips dataset, we present the performance of three detectors as a baseline to do the following: 1) elementarily summarize the difficulties of the dataset for ship detection; 2) show detection results for researchers using the dataset; and 3) make a comparison to identify the strengths and weaknesses of the baseline algorithms. In practice, the SeaShips dataset would hopefully advance research and applications on ship detection.

Large-Scale Study of Perceptual Video Quality.

The great variations of videographic skills in videography, camera designs, compression and processing protocols, communication and bandwidth environments, and displays leads to an enormous variety of video impairments. Current noreference (NR) video quality models are unable to handle this diversity of distortions. This is true in part because available video quality assessment databases contain very limited content, fixed resolutions, were captured using a small number of camera devices by a few videographers and have been subjected to a modest number of distortions. As such, these databases fail to adequately represent real world videos, which contain very different kinds of content obtained under highly diverse imaging conditions and are subject to authentic, complex and often commingled distortions that are difficult or impossible to simulate. As a result, NR video quality predictors tested on real-world video data often perform poorly. Towards advancing NR video quality prediction, we have constructed a largescale video quality assessment database containing 585 videos of unique content, captured by a large number of users, with wide ranges of levels of complex, authentic distortions. We collected a large number of subjective video quality scores via crowdsourcing. A total of 4776 unique participants took part in the study, yielding more than 205000 opinion scores, resulting in an average of 240 recorded human opinions per video. We demonstrate the value of the new resource, which we call the LIVE Video Quality Challenge Database (LIVE-VQC for short), by conducting a comparison of leading NR video quality predictors on it. This study is the largest video quality assessment study ever conducted along several key dimensions: number of unique contents, capture devices, distortion types and combinations of distortions, study participants, and recorded subjective scores. The database is available for download on this link: http://live.ece.utexas.edu/research/LIVEVQC/index.html.