Multiple Video Objects Research Articles

Cascade transformers with dynamic attention for video question answering

Visual question answering (VQA) has become a hot study topic with challenging motivation of correctly answering the videos or images questions in recent years. However, the existing VQA model mostly aimed at answering questions about images and performed poorly in the video question answering (VideoQA) domain. VideoQA needs to simultaneously consider the correlations between video frames and the dynamic information of multiple objects in video. Therefore, we propose a novel Cascade Transformers with Dynamic Attention for Video Question Answering (CTDA-QA), which aims to simultaneously solve the above considerations. Specifically, the proposed CTDA-QA model utilizes multiple transformers structure to encode videos for reasoning complex spatial and temporal information, which is different from the previous recurrent neural network methods. Besides, in order to effectively capture the dynamic information from various scenarios in videos, a flexible attention module has been proposed to explore the essential relations between objects in a dynamic timeline. Finally, to avoid spurious answers and fully explore the cross-modal relationships, a mixed-supervised learning strategy is designed for optimizing the reasoning tasks. The experiments on several benchmark video question–answer datasets clearly verify the performance and effectiveness of CTDA-QA, which contains the results in contrast to the state-of-the-art methods. Besides, the provided ablation study and visualization results further reveal the potential of CTDA-QA.

Real-time siamese multiple object tracker with enhanced proposals

Maintaining the identity of multiple objects in real-time video is a challenging task, as it is not always feasible to run a detector on every frame. Thus, motion estimation systems are often employed, which either do not scale well with the number of targets or produce features with limited semantic information. To solve the aforementioned problems and allow the tracking of dozens of arbitrary objects in real-time, we propose SiamMOTION. SiamMOTION includes a novel proposal engine that produces quality features through an attention mechanism and a region-of-interest extractor fed by an inertia module and powered by a feature pyramid network. Finally, the extracted tensors enter a comparison head that efficiently matches pairs of exemplars and search areas, generating quality predictions via a pairwise depthwise region proposal network and a multi-object penalization module. SiamMOTION has been validated on five public benchmarks, achieving leading performance against current state-of-the-art trackers. Code available at: https://www.github.com/lorenzovaquero/SiamMOTION

Robust tracking of multiple objects in video by adaptive fusion of subband particle filters

Tracking of moving objects in video sequences is an important research problem because of its many industrial, biomedical, and security applications. Significant progress has been made on this topic in the last few decades. However, the ability to track objects accurately in video sequences that have challenging conditions and unexpected events, e.g. background motion and shadows; objects with different sizes and contrasts; a sudden change in illumination; partial object camouflage; and low signal-to-noise ratio, remains an important research problem. To address such difficulties, the authors developed a robust multiscale visual tracker that represents a captured video frame as different subbands in the wavelet domain. It then applies N independent particle filters to a small subset of these subbands, where the choice of this subset of wavelet subbands changes with each captured frame. Finally, it fuses the outputs of these N independent particle filters to obtain final position tracks of multiple moving objects in the video sequence. To demonstrate the robustness of their multiscale visual tracker, they applied it to four example videos that exhibit different challenges. Compared to a standard full-resolution particle filter-based tracker and a single wavelet subband (LL)2-based tracker, their multiscale tracker demonstrates significantly better tracking performance.

DUAL-TREE COMPLEX WAVELET TRANSFORM FOR ROBUST VISUAL TRACKING OF MULTIPLE OBJECTS IN VIDEO

We developed a robust multiscale visual tracker of multiple objects in video using the Dual-tree Complex Wavelet Transform (DT-CWT). Real-valued wavelet transforms were previously used for visual tracking, but most suffer from shift variance and lack of directional selectivity. Therefore, we used DT-CWT to avoid such shortcomings. In our tracker, a captured video frame was represented as different subbands using DT-CWT. Then we applied N independent particle filters to a small subset of these subbands, where the choice of these subbands changed adaptively with each captured frame.Finally, we fused the position tracks resulting from these particle filters to obtain final position tracks of multiple moving objects in the video. To demonstrate robustness of our visual tracker, we compared the performance of our multiscale tracker to a standard particle filter full resolution-based tracker and a single wavelet subband (LL)2 based tracker, our multiscale tracker demonstrates significantly better tracking performance.

Category Level Object Discovery Using Dynamic Topic Model

Category level object discovery is important for a number of applications such as remote sensing image classification, and data mining in images and video sequences. This paper presents a novel unsupervised learning algorithm for discovering object category and their locations in video sequences. Both appearance consistency and motion consistency of local patches across frames are exploited. Video patches are first extracted and represented by spatial-temporal context words. A dynamic topic model is then introduced to learn object categories in video sequences. The proposed dynamic model can categorize and localize multiple objects in a single video. Experimental results on the CamVid dataset and the VISATTM dataset demonstrate the effectiveness of our method.

Anomalous video event detection using spatiotemporal context

Compared to other anomalous video event detection approaches that analyze object trajectories only, we propose a context-aware method to detect anomalies. By tracking all moving objects in the video, three different levels of spatiotemporal contexts are considered, i.e., point anomaly of a video object, sequential anomaly of an object trajectory, and co-occurrence anomaly of multiple video objects. A hierarchical data mining approach is proposed. At each level, frequency-based analysis is performed to automatically discover regular rules of normal events. Events deviating from these rules are identified as anomalies. The proposed method is computationally efficient and can infer complex rules. Experiments on real traffic video validate that the detected video anomalies are hazardous or illegal according to traffic regulations.

Discovering objects and their location in videos using spatial-temporal context words

We present a novel unsupervised learning algorithm for discovering objects and their location in videos from moving cameras. The videos can switch between different shots, and contain cluttered background, occlusion, camera motion, and multiple independently moving objects. We exploit both appearance consistency and spatial configuration consistency of local patches across frames for object recognition and localization. The contributions of this paper are twofold. First, we propose a combined approach for simultaneous spatial context and temporal context generation. Local video patches are extracted and described using the generated spatial-temporal context words. Second, a dynamic topic model, based on the representation of a bag of spatial-temporal context words, is introduced to learn object category models in video sequences. The proposed model can categorize and localize multiple objects in a single video. Objects leaving or entering the scene at multiple times can also be handled efficiently in the dynamic framework. Experimental results on the CamVid data set and the VISAT™ data set demonstrate the effectiveness and robustness of the proposed method.

Controlling the Bit Rate of Multi-Object Videos With Noncooperative Game Theory

This paper proposes an object-level rate control algorithm to jointly controlling the bit rates of multiple video objects. Utilizing noncooperative game theory, the proposed rate control algorithm mimics the behaviors of players representing video objects. Each player competes for available bits to optimize its visual quality. The algorithm finds an ?optimal solution? in that it conforms to the mixed strategy Nash equilibrium, which is the probability distribution of the actions carried by the players that maximizes their expected payoffs (the number of bits). The game is played iteratively, and the expected payoff of each play is accumulated. The game terminates when all of the available bits for the specific time instant have been distributed to video object planes (VOPs). The advantage of the proposed scheme is that the bidding objects divide the bits among themselves automatically and fairly, according to their encoding complexity, and with an overall solution that is strategically optimal under the given circumstances. To minimize buffer fluctuation and avoid buffer overflow and underflow, a proportional-integral-derivative (PID) control based buffer policy is utilized.

Optimized scalable cache management for video streaming system

An important research issue in video streaming is how to efficiently utilize the network resources to provide clients instant access to multiple video objects. Caching strategy and transmission scheme are the two essential points inside the video streaming framework. Recent research efforts on them are not sufficient due to their inflexible support for scalable encoded video streams and heterogeneous requests from clients. In this paper, we propose an optimized caching strategy (OCS) and a scalable transmission scheme (STS) for scalable coded video streaming. By exploring the characteristics of video streaming workload and system design objectives, OCS and STS work efficiently to minimize both network bandwidth cost and user access latency. Firstly, we analyze the caching problem for the proxy-assisted video streaming system and derive a maneuverable caching scenario. Secondly, we develop an efficient transmission scheme for scalable coded videos. Thirdly, we formulate a multi-objective optimization model with closed-form expressions to obtain the optimized caching strategy. Finally, with designed algorithms, an excellent compromise between two competing objectives (minimizing the bandwidth cost and the access latency) is achieved. We start our evaluation by studying the optimized caching strategy for a single video object. Then we apply the strategy to multiple video objects and illustrate the tradeoff between the optimization objectives. Our evaluation results show that compared with other caching strategies, the proposed optimized scalable caching strategy can achieve a significant reduction in bandwidth cost with even a small proxy cache size. Meanwhile, the best performance (in terms of bandwidth cost) is obtained together with the proposed scalable batch-patching transmission scheme.

Adaptive particle sampling and adaptive appearance for multiple video object tracking

In this work, we propose an innovative method to integrate the Kalman filter and adaptive particle sampling for multiple video object tracking. Taking advantage of both the closed-form equations for optimal prediction and update from Kalman filters and the versatility of particle sampling for measurement selection under occlusion or segmentation error cases, the proposed method achieves both high tracking accuracy and computational simplicity. The adaptive particle sampling, which uses parameters updated by Kalman filters, can thus require only a small number of particles to achieve high positioning and scaling accuracy. Also, the concept of adaptive appearance is applied to enhance the robustness of occlusion handling. The experimental results confirm the effectiveness of the proposed method.

Applying the multi-category learning to multiple video object extraction

Video object (VO) extraction is of great importance in multimedia processing. In recent years approaches have been proposed to deal with VO extraction as a classification problem. This type of methods calls for state-of-the-art classifiers because the performance is directly related to the accuracy of classification. Promising results have been reported for single object extraction using support vector machines (SVM) and its extensions. Multiple object extraction, on the other hand, still imposes great difficulty as multi-category classification is an ongoing research topic in machine learning. This paper introduces a new scheme of multi-category learning for multiple VO extraction, and demonstrates its effectiveness and advantages by experiments.

Dynamic Bit Allocation for Multiple Video Object Coding

<para xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> In MPEG-4, a visual scene may be treated as a composition of video objects and coded at object level. Such a flexible video coding framework makes it possible to code different video objects with different priority according to human perceptual characteristics. In this paper, we introduce a novel dynamic bit allocation framework to improve the subjective quality in such an object-based video coding system. We incorporate the rate distortion models with the dynamic priorities of the video objects and jointly encode video objects to minimize the weighted distortion within the bit budget constraint. We guarantee the human-interested video objects a better reconstructed quality by using the weighted bit allocation strategy in favour of the video objects with higher priority. To obtain the priority automatically, we apply a visual attention model. Comparing with traditional bit allocation algorithms, the objective quality of the object with higher priority is significantly improved under this framework. These results demonstrate the usefulness of this dynamic bit allocation framework. </para>

Segmentation and tracking of multiple video objects

This paper describes a technique that produces a content-based representation of a video shot composed by a background (still) mosaic and one or more foreground moving objects. Segmentation of moving objects is based on ego-motion compensation and on background modelling using tools from robust statistics. Region matching is carried out by an algorithm that operates on the Mahalanobis distance between region descriptors in two subsequent frames and uses singular value decomposition to compute a set of correspondences satisfying both the principle of proximity and the principle of exclusion. The sequence is represented as a layered graph, and specific techniques are introduced to cope with crossing and occlusion. Examples of MPEG-4 (main profile) encoding are reported.

Voting-based simultaneous tracking of multiple video objects

This paper proposes an automatic object tracking method based on both object segmentation and motion estimation for real-time content-oriented video applications. The method focuses on the issues of speed of execution and reliability in the presence of noise, coding artifacts, shadows, occlusion, and object split. Objects are tracked based on the similarity of their features in successive frames. This is done in three steps: feature extraction, object matching, and feature monitoring. In the first step, objects are segmented and their spatial and temporal features are computed. In the second step, using a nonlinear two-stage voting strategy, each object of the previous frame is matched with an object of the current frame creating a unique correspondence. In the third step, object changes, such objects occlusion or split, are monitored and object features are corrected. These new features are then used to update results of previous steps creating module interaction. The contributions in this paper are the real-time two-stage voting strategy, the monitoring of object changes to handle occlusion and object split, and the spatiotemporal adaptation of the tracking parameters. Experiments on indoor and outdoor video shots containing over 6000 frames, including deformable objects, multi-object occlusion, noise, and coding and object segmentation artifacts have demonstrated the reliability and real-time response of the proposed method.

Joint texture-shape optimization for MPEG-4 multiple video objects

In this paper, we present a uniform framework for optimal multiple video object bit allocation in MPEG-4. We combine the rate-distortion (R-D) models for the texture and shape information of arbitrarily shaped video objects to develop the joint texture-shape R-D models. The dynamic programming technique is applied to optimize the bit allocation for the multiple video objects. The simulation results demonstrate that the proposed joint texture-shape optimization algorithm outperforms the MPEG-4 verification model on the decoded picture quality.

Asynchronous rate control for multi-object videos

Object-based coding can potentially achieve a higher degree of compression and better visual quality. The objects in a scene are not always synchronous, that is, they have different temporal resolutions. In some applications, it is more efficient to transmit asynchronous objects with different temporal rates so as to achieve a better tradeoff between temporal and spatial resolutions. This requires to balance the qualities of individual objects while ensuring an overall visual quality with a given bit rate. This paper proposes a rate control algorithm for multiple video object encoding, which is suitable for both synchronous and asynchronous transmissions. The algorithm aims to maximize scene quality with an accurate bit rate, while efficiently handling buffer fullness. Using an efficient bit allocation strategy, the algorithm achieves accurate target bit rates, provides good coding quality, and decreases buffer overflow/underflow. The proposed algorithm also allows flexible priority adjustment among multiple objects to ensure overall better visual perception. Designed primarily for asynchronous objects, the algorithm treats the synchronous objects as a special case. Experimental results demonstrate that, when giving suitable asynchronous video object planes rates, the proposed algorithm achieves good temporal-spatial tradeoff while yielding accurate rate regulation and effective buffer control.

MPEG-4 video object-based rate allocation with variable temporal rates

In object-based coding, bit allocation is performed at the object level and temporal rates of different objects may vary. The proposed algorithm deals with these two issues when coding multiple video objects (MVOs). The proposed algorithm is able to successfully achieve the target bit rate, effectively code arbitrarily shaped MVOs with different temporal rates, and maintain a stable buffer level.

Fast and automatic video object segmentation and tracking for content-based applications

The new video-coding standard MPEG-4 enables content-based functionality, as well as high coding efficiency, by taking into account shape information of moving objects. A novel algorithm for segmentation of moving objects in video sequences and extraction of video object planes (VOPs) is proposed . For the case of multiple video objects in a scene, the extraction of a specific single video object (VO) based on connected components analysis and smoothness of VO displacement in successive frames is also discussed. Our algorithm begins with a robust double-edge map derived from the difference between two successive frames. After removing edge points which belong to the previous frame, the remaining edge map, moving edge (ME), is used to extract the VOP. The proposed algorithm is evaluated on an indoor sequence captured by a low-end camera as well as MPEG-4 test sequences and produces promising results.

Semantic video object extraction using four-band watershed and partition lattice operators

We conceive the problem of multiple semantic video object (SVO) extraction as an issue of designing extensive operators on a complete lattice of partitions. As a result, we propose a framework based on spatial partition generation and application of optimal operators on the generated partitions. Based on a statistical analysis of the watershed algorithm, we develop a multivalued morphological spatial segmentation method that incorporates an edge-driven marker extraction algorithm and a growing method which integrates both color and edge information. Having embedded the problem in the partition lattice framework, we propose a spatio-temporal regional maximum likelihood operator for extraction purposes. Some theoretical properties of the operator are established. Experimental results on several MPEG-4 test video sequences show that our scheme improves the precision of the extracted SVO boundaries compared to traditional watershed algorithms and provides accurate tracking of multiple SVOs in both static and moving camera scenarios. Furthermore, this scheme can be extended to deal with more general interactive video authoring systems.

Object-based transcoding for adaptable video content delivery

This paper introduces a new framework for video content delivery that is based on the transcoding of multiple video objects. Generally speaking, transcoding can be defined as the manipulation or conversion of data into another more desirable format. We consider manipulations of object-based video content, and more specifically, from one set of bit streams to another. Given the object-based framework, we present a set of new algorithms that are responsible for manipulating the original set of video bit streams. Depending on the particular strategy that is adopted, the transcoder attempts to satisfy network conditions or user requirements in various ways. One of the main contributions of this paper is to discuss the degrees of freedom within an object-based transcoder and demonstrate the flexibility that it has in adapting the content. Two approaches are considered: a dynamic programming approach and an approach that is based on available meta-data. Simulations with these two approaches provide insight regarding the bit allocation among objects and illustrates the tradeoffs that can be made in adapting the content. When certain meta-data about the content is available, we show that bit allocation can be significantly improved, key objects can be identified, and varying the temporal resolution of objects can be considered.