Previous Video Frames Research Articles

Efficient Motion Estimation for Remotely Controlled Vehicles: A Novel Algorithm Leveraging User Interaction

Unmanned Aerial Vehicles (UAVs) are increasingly being used in a variety of applications, including entertainment, surveillance, and delivery. However, the real-time Motion Estimation (ME) of UAVs is challenging due to the high speed and unpredictable movements of these vehicles. This paper presents a novel algorithm for optimizing ME for Remotely Controlled Vehicles (RCVs), with a particular focus on UAVs. The proposed algorithm, called Motion Dynamics Input Search (MDIS), incorporates information from vehicle motion dynamics estimation to enhance the accuracy and efficiency of ME. The MDIS algorithm addresses the challenges associated with real-time ME in RCVs by leveraging user input to guide the search for the most similar blocks in the previous video frame. Through extensive experimentation and evaluation, this study demonstrates the effectiveness of the proposed algorithm in improving ME performance for RCVs. The findings highlight the potential impact of user interaction and motion dynamics estimation in shaping the future of ME algorithms for RCVs and similar applications.

Cross-stage recurrent feature sharing network for video dehazing

Hazy weather conditions feed serious haze effects, such as color and visibility degradations in the captured videos. Extensive research on video dehazing has been carried out. However, they lack to balance the spatial and high-level contextual information during restoration. Being a complex task in this paper, we propose a cross-stage recurrent feature sharing network to balance the competing objective. In the proposed network, a cross-stage feature merging module operates on learned features of the previous video frame using the proposed encoder decoder based generator architecture. Learned features fed into the same encoder decoder framework for the current video frame feature learning. This module benefit the network while restoring the current video frame. It shares the feature information between two consecutive frames as they have a very minute information change. Meanwhile, the design of our encoder decoder generator architecture is based on a multi-receptive resolution module, feature fusion module, and pixel-wise spatial attention module. These modules mainly focus on acquiring broad contextual information, fusing intermediate-stage features to increase the learning ability of the network, and capturing robust features while discarding unnecessary features. Experimental evaluation results and ablation study show that the proposed network is superior to existing state-of-the-art methods for video dehazing.

High-Movement Human Segmentation in Video Using Adaptive N-Frames Ensemble

A wide range of camera apps and online video conferencing services support the feature of changing the background in real-time for aesthetic, privacy, and security reasons. Numerous studies show that the Deep-Learning (DL) is a suitable option for human segmentation, and the ensemble of multiple DL-based segmentation models can improve the segmentation result. However, these approaches are not as effective when directly applied to the image segmentation in a video. This paper proposes an Adaptive N-Frames Ensemble (AFE) approach for high-movement human segmentation in a video using an ensemble of multiple DL models. In contrast to an ensemble, which executes multiple DL models simultaneously for every single video frame, the proposed AFE approach executes only a single DL model upon a current video frame. It combines the segmentation outputs of previous frames for the final segmentation output when the frame difference is less than a particular threshold. Our method employs the idea of the N-Frames Ensemble (NFE) method, which uses the ensemble of the image segmentation of a current video frame and previous video frames. However, NFE is not suitable for the segmentation of fast-moving objects in a video nor a video with low frame rates. The proposed AFE approach addresses the limitations of the NFE method. Our experiment uses three human segmentation models, namely Fully Convolutional Network (FCN), DeepLabv3, and Mediapipe. We evaluated our approach using 1711 videos of the TikTok50f dataset with a single-person view. The TikTok50f dataset is a reconstructed version of the publicly available TikTok dataset by cropping, resizing and dividing it into videos having 50 frames each. This paper compares the proposed AFE with single models and the Two-Models Ensemble, as well as the NFE models. The experiment results show that the proposed AFE is suitable for low-movement as well as high-movement human segmentation in a video.

SLAM-OR: Simultaneous Localization, Mapping and Object Recognition Using Video Sensors Data in Open Environments from the Sparse Points Cloud.

In this paper, we propose a novel approach that enables simultaneous localization, mapping (SLAM) and objects recognition using visual sensors data in open environments that is capable to work on sparse data point clouds. In the proposed algorithm the ORB-SLAM uses the current and previous monocular visual sensors video frame to determine observer position and to determine a cloud of points that represent objects in the environment, while the deep neural network uses the current frame to detect and recognize objects (OR). In the next step, the sparse point cloud returned from the SLAM algorithm is compared with the area recognized by the OR network. Because each point from the 3D map has its counterpart in the current frame, therefore the filtration of points matching the area recognized by the OR algorithm is performed. The clustering algorithm determines areas in which points are densely distributed in order to detect spatial positions of objects detected by OR. Then by using principal component analysis (PCA)—based heuristic we estimate bounding boxes of detected objects. The image processing pipeline that uses sparse point clouds generated by SLAM in order to determine positions of objects recognized by deep neural network and mentioned PCA heuristic are main novelties of our solution. In contrary to state-of-the-art approaches, our algorithm does not require any additional calculations like generation of dense point clouds for objects positioning, which highly simplifies the task. We have evaluated our research on large benchmark dataset using various state-of-the-art OR architectures (YOLO, MobileNet, RetinaNet) and clustering algorithms (DBSCAN and OPTICS) obtaining promising results. Both our source codes and evaluation data sets are available for download, so our results can be easily reproduced.

Real-Time Shadow Detection From Live Outdoor Videos for Augmented Reality.

Simulating shadow interactions between real and virtual objects is important for augmented reality (AR), in which accurately and efficiently detecting real shadows from live videos is a crucial step. Most of the existing methods are capable of processing only scenes captured under a fixed viewpoint. In contrast, this article proposes a new framework for shadow detection in live outdoor videos captured under moving viewpoints. The framework splits each frame into a tracked region, which is the region tracked from the previous video frame through optical flow analysis, and an emerging region, which is newly introduced into the scene due to the moving viewpoint. The framework subsequently extracts features based on the intensity profiles surrounding the boundaries of candidate shadow regions. These features are then utilized to both correct erroneous shadow boundaries for the tracked region and to detect shadow boundaries for the emerging region by a Bayesian learning module. To remove spurious shadows, spatial layout constraints are further considered for emerging regions. The experimental results demonstrate that the proposed framework outperforms the state-of-the-art shadow tracking and detection algorithms on a variety of challenging cases in real time, including shadows on backgrounds with complex textures, nonplanar shadows, fast-moving shadows with changing typologies, and shadows cast by nonrigid objects. The quantitative experiments show that our method outperforms the best existing method, achieving a 33.3% increase in the average Fmeasure on a self-collected database. Coupled with an image-based shadow-casting method, the proposed framework generates realistic shadow interaction results. This capability will be particularly beneficial for supporting AR applications.

Measuring Traffic Volumes Using an Autoencoder with No Need to Tag Images with Labels

Almost all vision technologies that are used to measure traffic volume use a two-step procedure that involves tracking and detecting. Object detection algorithms such as YOLO and Fast-RCNN have been successfully applied to detecting vehicles. The tracking of vehicles requires an additional algorithm that can trace the vehicles that appear in a previous video frame to their appearance in a subsequent frame. This two-step algorithm prevails in the field but requires substantial computation resources for training, testing, and evaluation. The present study devised a simpler algorithm based on an autoencoder that requires no labeled data for training. An autoencoder was trained on the pixel intensities of a virtual line placed on images in an unsupervised manner. The last hidden node of the former encoding portion of the autoencoder generates a scalar signal that can be used to judge whether a vehicle is passing. A cycle-consistent generative adversarial network (CycleGAN) was used to transform an original input photo of complex vehicle images and backgrounds into a simple illustration input image that enhances the performance of the autoencoder in judging the presence of a vehicle. The proposed model is much lighter and faster than a YOLO-based model, and accuracy of the proposed model is equivalent to, or better than, a YOLO-based model. In measuring traffic volumes, the proposed approach turned out to be robust in terms of both accuracy and efficiency.

DevsNet: Deep Video Saliency Network using Short-term and Long-term Cues

Recently, there have been various saliency detection methods proposed for still images based on deep learning techniques. However, the research on saliency detection for video sequences is still limited. In this study, we introduce a novel deep learning framework of saliency detection for video sequences, namely Deep Video Saliency Network (DevsNet). DevsNet mainly consists of two components: 3D Convolutional Network (3D-ConvNet) and Bidirectional Convolutional Long-Short Term Memory Network (B-ConvLSTM). 3D-ConvNet is constructed to learn short-term spatiotemporal information and the long-term spatiotemporal features are learned by B-ConvLSTM. The designed B-ConvLSTM can extract the temporal information not just from the previous video frames but also from the next frames, which demonstrates that the proposed model considers both the forward and backward temporal information. By combining the short-term and long-term spatiotemporal cues, the proposed DevsNet can extract saliency information for video sequences effectively and efficiently. Extensive experiments have been conducted to show that the proposed model can obtain better performance in spatiotemporal saliency prediction than the state-of-the-art models.

Autonomous lane detection on hilly terrain for perception-based navigation applications

ABSTRACTThe objective of our venture is to actualize a lane detection framework considering expanding the usability of driver assistance system functions, attempting to overcome the problems of the generalized method. It is tuned to detect mountainous roads on the Indian hilly terrain. The proposed technique acquires a video sequence as input and performs operation on each frame to extract the candidate lane lines. The novelty of this work is that a unique method is used to decide upon whether the lane is on the straight or curved roads and adaptive thresholding is used as it can adjust its value for various environmental conditions. Kalman filter is used in the lane tracking stage where the feasible results obtained for the present frame are compared with the results in the previous video frames. Comparison of the presented algorithm with the conventional method is carried out and attained an average detection accuracy of 93%.

Long-term tracking algorithm using deep features and a single shot multibox detector

The ability for computers to target and track objects within video sequences has attracted increasing amounts of attention in recent years. Many tracking algorithms adopt correlation filtering-based algorithms, in which targeting of features in the previous video frame is essential to predicting the target’s position in the next frame. A serious problem in tracking is drift. If drift occurs in the present frame, it becomes increasingly difficult to correct this error in later frames, which can lead to loss of the tracked target and greatly reduced tracking accuracy and stability. To solve this problem and improve tracking performance, we propose an algorithm that incorporates deep features. A sparse representation method is used to filter deep features and improve their complexity. Then, a Siamese network is used to judge the similarities between the target and template and determine a confidence level. Next, a single shot multibox detector (SSD) is introduced to support tracking, which is activated when the confidence level falls below a certain threshold. The correct target can be found by scanning the present frame using the detector and then updating the template. The confidence level is then reduced and the SSD continues working to resolve the drift. Tracking stability and accuracy are greatly improved after the redetection process. We use large-scale benchmark datasets to demonstrate that the proposed approach performs better than existing state-of-the-art approaches.

Low Complexity Multimedia Encryption

Selective encryption algorithms have been proposed to encrypt syntax elements such as intra prediction modes, the sign bit of nonzero DCT coefficients, along with the sign bit of motion vectors. These syntax elements are sensitive enough to produce effective scrambling effect with a relative low computational cost. In this paper, a novel scheme is proposed to further optimize the computational overhead incurred by the encryption for energy critical multimedia applications. The proposed scheme adjusts the selection of syntax elements to be encrypted according to the scene transitions within adjacent video frames. The ratio of intra-coded macroblocks in inter (P and B) frames is calculated and compared with an adaptive threshold value to detect the scene transitions. Furthermore, based on statistical analysis for a few video sequences, a dynamic threshold model to detect the scene transition is proposed. When there is a scene transition between the previous video frame and the current video frame, intra prediction modes and the sign bit of DCT coefficients in the current frame are chosen as syntax elements to be encrypted, whereas in the absence of a scene transition, the sign bit of motion vectors is chosen as the only sensitive syntax elements to be encrypted. Experimental results show that compared with previous work in this field, the proposed scheme can efficiently lower the computational cost incurred by the encryption while maintaining a similar perceptual scrambling effect.

Spatio-Temporal Flame Modeling and Dynamic Texture Analysis for Automatic Video-Based Fire Detection

Every year, a large number of wildfires all over the world burn forested lands, causing adverse ecological, economic, and social impacts. Beyond taking precautionary measures, early warning and immediate response are the only ways to avoid great losses. To this end, in this paper we propose a computer vision approach for fire-flame detection to be used by an early-warning fire monitoring system. Initially, candidate fire regions in a frame are defined using background subtraction and color analysis based on a nonparametric model. Subsequently, the fire behavior is modeled by employing various spatio-temporal features, such as color probability, flickering, spatial, and spatio-temporal energy, while dynamic texture analysis is applied in each candidate region using linear dynamical systems and a bag-of-systems approach. To increase the robustness of the algorithm, the spatio-temporal consistency energy of each candidate fire region is estimated by exploiting prior knowledge about the possible existence of fire in neighboring blocks from the current and previous video frames. As a final step, a two-class support vector machine classifier is used to classify the candidate regions. Experimental results have shown that the proposed method outperforms existing state-of-the-art algorithms.

Top-Down and Bottom-Up Cues Based Moving Object Detection for Varied Background Video Sequences

Moving object detection is a crucial and critical task for any surveillance system. Conventionally, a moving object detection task is performed on the basis of consecutive frame difference or background models which are based on some mathematical aspects or probabilistic approaches. But, these approaches are based on some initial conditions and short amount of time is needed to learn all these models. Also, the bottleneck in all these previous approaches is that they require neat and clean background or need to create a background first by using some approaches and that it is essential to update them regularly to cope with the illuminating changes. In this paper, moving object detection is executed using visual attention where there is no need for background formulation and updates as it is background independent. Many bottom-up approaches and one combination of bottom-up and top-down approaches are proposed in the present paper. The proposed approaches seem more efficient due to inessential requirement of learning background model and due to being independent of previous video frames. Results indicate that the proposed approach works even against slight movements in the background and in various outdoor conditions.

Visual Object Tracking Based on Local Steering Kernels and Color Histograms

In this paper, we propose a visual object tracking framework, which employs an appearance-based representation of the target object, based on local steering kernel descriptors and color histogram information. This framework takes as input the region of the target object in the previous video frame and a stored instance of the target object, and tries to localize the object in the current frame by finding the frame region that best resembles the input. As the object view changes over time, the object model is updated, hence incorporating these changes. Color histogram similarity between the detected object and the surrounding background is employed for background subtraction. Experiments are conducted to test the performance of the proposed framework under various conditions. The proposed tracking scheme is proven to be successful in tracking objects under scale and rotation variations and partial occlusion, as well as in tracking rather slowly deformable articulated objects.

Error Concealment for Scalable Motion-Compensated Subband/Wavelet Video Coders

<para xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> In this paper, we present two error-concealment algorithms developed for scalable motion-compensated subband/wavelet video coders. These algorithms exploit the properties of motion-compensated temporal filtering to recover lost video data by motion compensation from correctly received previous and future video frames. Our experiments indicate that backward motion-compensated prediction outperforms replacement from neighboring correctly received frames by up to 3 dB in terms of PSNR. In addition, a bidirectional algorithm tops the unidirectional one by up to 1 dB. Also, visual improvements are often higher that PSNR improvements would suggest. </para>

Shadow resistant tracking using inertia constraints

In this paper, we present a new method for tracking objects with shadows. Traditional motion-based tracking schemes cannot usually distinguish the shadow from the object itself, and this results in a falsely captured object shape. If we want to utilize the object's shape information for a pattern recognition task, this poses a severe difficulty. In this paper we present a color processing scheme to project the image into an illumination invariant space such that the shadow's effect is greatly attenuated. The optical flow in this projected image together with the original image is used as a reference for object tracking so that we can extract the real object shape in the tracking process. We present a modified snake model for general video object tracking. Two new external forces are introduced into the snake equation based on the predictive contour and a new chordal string shape descriptor such that the active contour is attracted to a shape similar to the one in the previous video frame. The proposed method can deal with the problem of an object's ceasing movement temporarily, and can also avoid the problem of the snake tracking into the object interior. Global affine motion estimation is applied to mitigate the effect of camera motion, and hence the method can be applied in a general video environment. Experimental results show that the proposed method can track the real object even if there is strong shadow influence.

The use of attention and spatial information for rapid facial recognition in video

Bottom-up visual attention allows primates to quickly select regions of an image that contain salient objects. In artificial systems, restricting the task of object recognition to these regions allows faster recognition and unsupervised learning of multiple objects in cluttered scenes. A problem with this approach is that objects superficially dissimilar to the target are given the same consideration in recognition as similar objects. In video, objects recognized in previous frames at locations distant to the current fixation point are given the same consideration in recognition as objects previously recognized in locations closer to the current target of attention. Due to the continuity of smooth motion, objects recently recognized in previous frames at locations close to the current focus of attention have a high probability of matching the current target. Here we investigate rapid pruning of the facial recognition search space using the already-computed low-level features that guide attention and spatial information derived from previous video frames. For each video frame, Itti & Koch's bottom-up visual attention algorithm is used to select salient locations based on low-level features such as contrast, orientation, color, intensity, flicker and motion. This algorithm has shown to be highly effective in selecting faces as salient objects. Lowe's SIFT object recognition algorithm then extracts a signature of the attended object, for comparison with the facial database. The database search is prioritized for faces which better match the low-level features used to guide attention to the current candidate for recognition or those that were previously recognized near the current candidate's location. The SIFT signatures of the prioritized faces are then checked against the attended candidate for a match. By comparing performance of Lowe's recognition algorithm and Itti & Koch's bottom-up attention model with or without search space pruning we demonstrate that our pruning approach improves the speed of facial recognition in video footage.