Objects In Consecutive Frames Research Articles

Exploring Point-BEV Fusion for 3D Point Cloud Object Tracking with Transformer.

With the prevalent use of LiDAR sensors in autonomous driving, 3D point cloud object tracking has received increasing attention. In a point cloud sequence, 3D object tracking aims to predict the location and orientation of an object in consecutive frames. Motivated by the success of transformers, we propose Point Tracking TRansformer (PTTR), which efficiently predicts high-quality 3D tracking results in a coarse-to-fine manner with the help of transformer operations. PTTR consists of three novel designs. 1) Instead of random sampling, we design Relation-Aware Sampling to preserve relevant points to the given template during subsampling. 2) We propose a Point Relation Transformer for effective feature aggregation and feature matching between the template and search region. 3) Based on the coarse tracking results, we employ a novel Prediction Refinement Module to obtain the final refined prediction through local feature pooling. In addition, motivated by the favorable properties of the Bird's-Eye View (BEV) of point clouds in capturing object motion, we further design a more advanced framework named PTTR++, which incorporates both the point-wise view and BEV representation to exploit their complementary effect in generating high-quality tracking results. PTTR++ substantially boosts the tracking performance on top of PTTR with low computational overhead. Extensive experiments over multiple datasets show that our proposed approaches achieve superior 3D tracking accuracy and efficiency. Code will be available at https://github.com/Jasonkks/PTTR.

Anchor-free deep convolutional neural network for tracking and counting cotton seedlings and flowers

Accurate counting of plants and their organs in natural environments is essential for breeders and growers. For breeders, counting plants during the seedling stage aids in selecting genotypes with superior emergence rates, while for growers, it informs decisions about potential replanting. Meanwhile, counting specific plant organs, such as flowers, forecasts yields for different genotypes, offering insights into production levels. The overall goal of this study was to investigate a deep convolutional neural network-based tracking method, CenterTrack, for cotton seedling and flower counting from video frames. The network is extended from a customized CenterNet, which is an anchor-free object detector. CenterTrack predicts the detections of the current frame and displacements of detections between the previous frame and the current frame, which are used to associate the same object in consecutive frames. The modified CenterNet detector achieved high accuracy on both seedling and flower datasets with an overall AP50 of 0.962. The video tracking hyperparameters were optimized for each dataset using orthogonal tests. Experimental results showed that seedling and flower counts with optimized hyperparameters highly correlated with those of manual counts (R2 = 0.98 andR2 = 0.95) and the mean relative errors of 75 cotton seedling testing videos and 50 flower testing videos were 5.5 % and 10.8 %, respectively. An average counting speed of 20.4 frames per second was achieved with an input resolution of 1920 × 1080 pixels for both seedling and flower videos. The anchor-free deep convolution neural network-based tracking method provides automatic tracking and counting in video frames, which will significantly benefit plant breeding and crop management.

Object-Level Semantic and Velocity Feedback for Dynamic Occupancy Grids

LiDAR-based frameworks combining dynamic occupancy grids and object-level tracking are a popular approach for perception of the environment in autonomous driving applications. This paper presents a novel backchannel from the object-level module to the grid-level module that procures the enhancement of overall performance. This feedback leads to an enhanced grid representation by the inclusion of two new steps that allow semantic classification of the occupied space and the improvement of the dynamic estimation. To this end, objects extracted from the grid are analyzed with respect to potential object classes and displacement. Class likelihoods are filtered over time at cell-level using particles and a naive Bayesian classifier. The displacement information is computed taking into account semantic information and comparing objects in consecutive frames. Then, it is used to obtain velocity measurements that are used to enhance grid's dynamic estimation. In contrast to other approaches in the literature seeking similar objectives, this proposal does not rely on additional sensing technologies or neural networks. The evaluation is conducted with real sensor data in challenging urban scenarios.

A Rough Set Framework for Multihuman Tracking in Surveillance Video

Multi-human tracking has become a key focus area in video surveillance applications. Several research works have been carried out in multi-human tracking in the past two decades. The techniques such as kalman filter, particle filter, markov chain process, blob detection, etc., have been used to detect and track humans. However, tracking the human objects in consecutive frames is still a challenging problem due to spatial disorder, non-linear motion and occlusion of human objects. Also, the human object labeling becomes difficult since most of the similarity measures used in the classification process do not consider the positional coordinates while computing the similarity. This paper addresses the above challenges by introducing a rough set framework which identifies the human objects using modified bounding box generation technique and by applying the rough set classifier. The framework was tested on the benchmark dataset PETS09 and the experimental results were compared with OCELM, MODT and MPTS algorithms. The comparative analysis shows that the rough set framework outperforms the existing algorithms in terms of detection and tracking accuracy even in the cases of overlapped and hidden human objects.

Modified YOLO Module for Efficient Object Tracking in a Video

In the proposed work, initially, the YOLO algorithm is used to extract and classify objects in a frame. In the sequence of frames, due to various reasons the confidence measure suddenly drops. This changes the class of an object in consecutive frames which affects the object tracking and counting process severely. To overcome this limitation of the YOLO algorithm, it is modified to enable and track the same object efficiently in the sequence of frames. This will in turn increase object tracking and counting accuracy. In the proposed work drastic change in confidence scores and class change of an object in consecutive frames are identified by tracking the confidence of a particular object in the sequence of frames. These outliers are detected and removed using the RANSAC algorithm. After the removal of the outliers, interpolation is applied to get the new confidence score at that point. By applying the proposed method a smooth confidence measure variation is obtained across the frames. Using this, average counting accuracy has been increased from 66 % to 87 % and overall average object classification accuracy is in the range of 94 - 96 % for various standard dataset.

Boosting UAV Tracking With Voxel-Based Trajectory-Aware Pre-Training

Siamese network-based object tracking has remarkably promoted the automatic capability for highly-maneuvered unmanned aerial vehicles (UAVs). However, the leading-edge tracking framework often depends on template matching, making it trapped when facing multiple views of object in consecutive frames. Moreover, the general image-level pretrained backbone can overfit to holistic representations, causing the misalignment to learn object-level properties in UAV tracking. To tackle these issues, this work presents <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">TRTrack</i> , a comprehensive framework to fully exploit the stereoscopic representation for UAV tracking. Specifically, a novel pre-training paradigm method is proposed. Through trajectory-aware reconstruction training (TRT), the capability of the backbone to extract stereoscopic structure feature is strengthened without any parameter increment. Accordingly, an innovative hierarchical self-attention Transformer is proposed to capture the local detail information and global structure knowledge. For optimizing the correlation map, we proposed a novel spatial correlation refinement (SCR) module, which promotes the capability of modeling the long-range spatial dependencies. Comprehensive experiments on three UAV challenging benchmarks demonstrate that the proposed TRTrack achieves superior UAV tracking performance in both precision and efficiency. Quantitative tests in real-world settings fully prove the effectiveness of our work.

DGM-VINS: Visual–Inertial SLAM for Complex Dynamic Environments With Joint Geometry Feature Extraction and Multiple Object Tracking

Most current state-of-the-art simultaneous localization and mapping (SLAM) algorithms perform well in static environments. However, their applications in real-world scenarios are limited by the assumption that environments are static because their performance becomes unstable in complex dynamic environments. To enhance system stability and localization accuracy in complex dynamic scenes, this article presents a novel visual-inertial SLAM system called DGM-VINS. In DGM-VINS, a joint geometric dynamic feature extraction module (JGDFE) is designed that can combine the advantages of multiple geometric constraints and effectively reduce the limitations of a single geometric constraint in the application process. In addition, a temporal instance segmentation module (TISM) is presented to establish the temporal correlation of instance objects in consecutive frames, which effectively addresses the instance segmentation issue in complex environments. The inertial measurement unit (IMU) is utilized for motion prediction and consistency detection to improve localization accuracy in challenging environments with weak textures. The proposed methodology is tested in various public datasets and actual scenarios, and the results demonstrate superior accuracy and robustness than existing methods in complex dynamic scenarios.

Spatiotemporal tubelet feature aggregation and object linking for small object detection in videos

This paper addresses the problem of exploiting spatiotemporal information to improve small object detection precision in video. We propose a two-stage object detector called FANet based on short-term spatiotemporal feature aggregation and long-term object linking to refine object detections. First, we generate a set of short tubelet proposals. Then, we aggregate RoI pooled deep features throughout the tubelet using a new temporal pooling operator that summarizes the information with a fixed output size independent of the tubelet length. In addition, we define a double head implementation that we feed with spatiotemporal information for spatiotemporal classification and with spatial information for object localization and spatial classification. Finally, a long-term linking method builds long tubes with the previously calculated short tubelets to overcome detection errors. The association strategy addresses the generally low overlap between instances of small objects in consecutive frames by reducing the influence of the overlap in the final linking score. We evaluated our model in three different datasets with small objects, outperforming previous state-of-the-art spatiotemporal object detectors and our spatial baseline.

An Effectively Finite-Tailed Updating for Multiple Object Tracking in Crowd Scenes

Multiple Object Tracking (MOT) focuses on tracking all the objects in a video. Most MOT solutions follow a tracking-by-detection or a joint detection tracking paradigm to generate the object trajectories by exploiting the correlations between the detected objects in consecutive frames. However, according to our observations, considering only the correlations between the objects in the current frame and the objects in the previous frame will lead to an exponential information decay over time, thus resulting in a misidentification of the object, especially in scenes with dense crowds and occlusions. To address this problem, we propose an effectively finite-tailed updating (FTU) strategy to generate the appearance template of the object in the current frame by exploiting its local temporal context in videos. To be specific, we model the appearance template for the object in the current frame on the appearance templates of the objects in multiple earlier frames and dynamically combine them to obtain a more effective representation. Extensive experiments have been conducted, and the experimental results show that our tracker outperforms the state-of-the-art methods on MOT Challenge Benchmark. We have achieved 73.7% and 73.0% IDF1, and 46.1% and 45.0% MT on the MOT16 and MOT17 datasets, which are 0.9% and 0.7% IDFI higher, and 1.4% and 1.8% MT higher than FairMOT repsectively.

Learnable spatiotemporal feature pyramid for prediction of future optical flow in videos

The success of deep learning-based techniques in solving various computer vision problems motivated the researchers to apply deep learning to predict the optical flow of a video in the next frame. However, the problem of predicting the motion of an object in the next few frames remains an unsolved and less explored problem. Given a sequence of frames, predicting the motion in the next few frames of the video becomes difficult in cases where the displacement of optical flow vector across frames is large. Traditional CNNs often fail to learn the dynamics of the objects across frames in case of large displacements of objects in consecutive frames. In this paper, we present an efficient CNN based on the concept of feature pyramid for extracting the spatial features from a few consecutive frames. The spatial features extracted from consecutive frames by a modified PWC-Net architecture are fed into a bidirectional LSTM for obtaining the temporal features. The proposed spatiotemporal feature pyramid is able to capture the abrupt motion of the moving objects in video, especially when displacement of the object is large across the consecutive frames. Further, the proposed spatiotemporal pyramidal feature can effectively predict the optical flow in next few frames, instead of predicting only the next frame. The proposed method of predicting optical flow outperforms the state of the art when applied on challenging datasets such as “MPI Sintel Final Pass,” “Monkaa” and “Flying Chairs” where abrupt and large displacement of the moving objects in consecutive frames is the main challenge.

A Novel Vehicle Tracking ID Switches Algorithm for Driving Recording Sensors.

The main task for real-time vehicle tracking is establishing associations with objects in consecutive frames. After occlusion occurs between vehicles during the tracking process, the vehicle is given a new ID when it is tracked again. In this study, a novel method to track vehicles between video frames was constructed. This method was applied on driving recorder sensors. The neural network model was trained by YOLO v3 and the system collects video of vehicles on the road using a driving data recorder (DDR). We used the modified Deep SORT algorithm with a Kalman filter to predict the position of the vehicles and to calculate the Mahalanobis, cosine, and Euclidean distances. Appearance metrics were incorporated into the cosine distances. The experiments proved that our algorithm can effectively reduce the number of ID switches by 29.95% on the model trained on the BDD100K dataset, and it can reduce the number of ID switches by 32.16% on the model trained on the COCO dataset.

Parallelization of the Honeybee Search Algorithm for Object Tracking

Object tracking refers to the relocation of specific objects in consecutive frames of a video sequence. Presently, this visual task is still considered an open research issue, and the computer science community attempted solutions from the standpoint of methodologies, algorithms, criteria, benchmarks, and so on. This article introduces a GPU-parallelized swarm algorithm, called the Honeybee Search Algorithm (HSA), which is a hybrid algorithm combining swarm intelligence and evolutionary algorithm principles, and was previously designed for three-dimensional reconstruction. This heuristic inspired by the search for food of honeybees, and here adapted to the problem of object tracking using GPU parallel computing, is extended from the original proposal of HSA towards video processing. In this work, the normalized cross-correlation (NCC) criteria is used as the fitness function. Experiments using 314 video sequences of the ALOV benchmark provides evidence about the quality regarding tracking accuracy and processing time. Also, according to these experiments, the proposed methodology is robust to high levels of Gaussian noise added to the image frames, and this confirms that the accuracy of the original NCC is preserved with the advantage of acceleration, offering the possibility of accelerating latest trackers using this methodology.

Recovery of Lost Color and Depth Frames in Multiview Videos.

In this paper, we consider an integrated error concealment system for lost color frames and lost depth frames in multiview videos with depths. We first proposed a pixel-based color error-concealment method with the use of depth information. Instead of assuming that the same moving object in consecutive frames has minimal depth difference, as is done in a state-of-the-art method, a more realistic situation in which the same moving object in consecutive frames can be in different depths is considered. In the derived motion vector candidate set, we consider all the candidate motion vectors in the set, and weight the reference pixels by the depth differences to obtain the final recovered pixel. Compared with the two state-of-the-art methods, the proposed method has average peak signal-to-noise ratio gains of up to 8.73 and 3.98 dB, respectively. Second, we proposed an iterative depth frame error-concealment method. The initial recovered depth frame is obtained by depth-image-based rendering from another available view. The holes in the recovered depth frame are then filled in the proposed priority order. Preprocessing methods (depth difference compensation and inconsistent pixel removal) are performed to improve the performance. Compared with a method that uses the available motion vector in a color frame to recover the lost depth pixels, the hybrid motion vector extrapolation method, the inpainting method and the proposed method have gains of up to 4.31, 10.29, and 6.04 dB, respectively. Finally, for the situation in which the color and the depth frames are lost at the same time, our two methods jointly perform better with a gain of up to 7.79 dB.

Confidence-Based Data Association and Discriminative Deep Appearance Learning for Robust Online Multi-Object Tracking.

Online multi-object tracking aims at estimating the tracks of multiple objects instantly with each incoming frame and the information provided up to the moment. It still remains a difficult problem in complex scenes, because of the large ambiguity in associating multiple objects in consecutive frames and the low discriminability between objects appearances. In this paper, we propose a robust online multi-object tracking method that can handle these difficulties effectively. We first define the tracklet confidence using the detectability and continuity of a tracklet, and decompose a multi-object tracking problem into small subproblems based on the tracklet confidence. We then solve the online multi-object tracking problem by associating tracklets and detections in different ways according to their confidence values. Based on this strategy, tracklets sequentially grow with online-provided detections, and fragmented tracklets are linked up with others without any iterative and expensive association steps. For more reliable association between tracklets and detections, we also propose a deep appearance learning method to learn a discriminative appearance model from large training datasets, since the conventional appearance learning methods do not provide rich representation that can distinguish multiple objects with large appearance variations. In addition, we combine online transfer learning for improving appearance discriminability by adapting the pre-trained deep model during online tracking. Experiments with challenging public datasets show distinct performance improvement over other state-of-the-arts batch and online tracking methods, and prove the effect and usefulness of the proposed methods for online multi-object tracking.

Moving Object Tracking Based on Gaussian Kernel and Template Modelling

The Project presents object tracking from videos based on template matching using Gaussian kernel with probability distribution function and mean shift algorithm. Initially, the target will be selected from chosen video sequence to track desired object in consecutive frames. The target will be utilized to determine the probability distribution function for similarity measurement between target and current processing frames. Here, Gaussian kernel function and its gradient are used here to find the PDF for corresponding templates. Similarity between two different images will be measured by weighted sum of Gaussian coefficients and PDFS. Mean shift approach used here to shifting the starting coordinates of template to find its similar features in consecutive frames to detect desired objects. The dissimilarity between the target model and target candidates will be expressed by a metric derived from Bhattacharyya coefficient. The project simulated results shows that moving object from video will be tracked accurately at different position and shape with help of templates in a considerable amount of time.

Design and Implementation of TrackingSystem

Locating an object of interest in continuous video frames can be termed as Object Tracking. Computer Vision is branch recently focussing on interaction between human and computer system. Continuously tracking desired object in consecutive frames is a difficult task. Tracking helps in sorting out situations such criminal incidents, traffic congestion, reckless driving and speed violation. The object in motion is detected using algorithm based on frame differencing and background subtraction. Morphological operations along with filtering are implemented to eliminate noise. The object co-ordinates are displayed along with direction of motion. It is a real time human tracking System. The same algorithm is used for color (yellow) based tracking. For vehicle tracking Simulink model is implemented in which stored video is processed using frame differencing and optical flow. Thus using this algorithm continuous monitoring and surveillance system is introduced.

Probabilistic motion pixel detection for the reduction of ghost artifacts in high dynamic range images from multiple exposures

Abstract This paper presents an algorithm for compositing a high dynamic range (HDR) image from multi-exposure images, considering inconsistent pixels for the reduction of ghost artifacts. In HDR images, ghost artifacts may appear when there are moving objects while taking multiple images with different exposures. To prevent such artifacts, it is important to detect inconsistent pixels caused by moving objects in consecutive frames and then to assign zero weights to the corresponding pixels in the fusion process. This problem is formulated as a binary labeling problem based on a Markov random field (MRF) framework, the solution of which is a binary map for each exposure image, which identifies the pixels to be excluded in the fusion process. To obtain the ghost map, the distribution of zero-mean normalized cross-correlation (ZNCC) of an image with respect to the reference frame is modeled as a mixture of Gaussian functions, and the parameters of this function are used to design the energy function. However, this method does not well detect faint objects that are in low-contrast regions due to over- or under-exposure, because the ZNCC does not show much difference in such areas. Hence, we obtain an additional ghost map for the low-contrast regions, based on the intensity relationship between the frames. Specifically, the intensity mapping function (IMF) between the frames is estimated using pixels from high-contrast regions without inconsistent pixels, and pixels out of the tolerance range of the IMF are considered moving pixels in the low-contrast regions. As a result, inconsistent pixels in both the low- and high-contrast areas are well found, and thus, HDR images without noticeable ghosts can be obtained.

해양환경에서 강건한 물표 추적 알고리즘

본 논문에서는 해양환경에서 취득한 열상 영상에서 물표를 강건하게 탐지하고 추적하는 알고리즘을 제안한다. 먼저 2-D 이산 Harr 웨이블렛 변환(DHWT) 기법을 이용하여 수평, 수직에지를 얻은 다음 수직 및 수평 에지들을 서로 곱하여 하나의 영상으로 결합해 돌출지도를 생성한다. 그런 다음 돌출지도를 이진화하여 물표를 추출한다. 물표를 추적하기 위하여 인접한 프레임에 존재하는 물체간의 가중치가 부여된 유클리디언 거리를 정합척도로 사용하였으며, 정합결과에 대해 물체의 나타남, 사라짐, 잘못된 물체 추출 등을 고려한 궤적관리를 통하여 최종적인 물체 궤적을 얻는다. 실험결과는 제안하는 알고리즘이 물체를 강건하게 추적함을 보인다. In this paper, we proposed a robust object extraction and tracking algorithm in the IR image sequence acquired in the sea environment. In order to extract size-invariant object, we detect horizontal and vertical edges by using DWT and combine it to generate saliency map. To extract object region, binarization technique is applied to saliency map. The correspondences between objects in consecutive frames are defined by the calculating minimum weighted Euclidean distance as a matching measure. Finally, object trajectories are determined by considering false correspondences such as entering object, vanishing objects and false object and so on. The proposed algorithm can find trajectories robustly, which has shown by experimental results.

ENHANCED TARGET TRACKING BASED ON MEAN SHIFT ALGORITHM FOR SATELLITE IMAGERY

Target tracking in high resolution satellite images is challenging task for computer vision field. In this paper we have proposed a mean shift algorithm based enhanced target tracking system for high resolution satellite imagery. In p roposed tracking algorithm, Target modeling is done using spectral features of target object i.e. Mean & Energy density function. Feature Vector space with minimum Euclidean Distance is used for predicting n ext possible position of target object in consecuti ve frames. Proposed tracking algorithm has been tested using two high resolution databases i.e. Harbor & Airport region database ac quired by WorldView-2 satellite at different times. Recall, Precision & F 1 score etc. performance parameters are also calcula ted for showing the tracking ability of the proposed method in real-time applica tions and are compared with the results of Regional Operator Design based tracking algorithm proposed in [1]. The results sh ow that our proposed method gives relatively better performance than the other tracking algorithms used in satellite imagery.

Development of a target recognition and following system for a field robot

The objective of this study was to develop a target recognition and following system for a field robot. The platform used in this study was an electric vehicle that had been modified into an automatic vehicle. A 3D camera was attached in the front of the vehicle to deliver depth and grayscale information on the environment. The degrees of correspondence between objects present in consecutive frames were calculated using their spatial descriptors and grayscale features. A correspondence assigner was designed to recognize the same objects in consecutive frames, and their locations could be acquired at the same time. Based on a pursuit strategy, an appropriate steering angle was calculated for the robot vehicle to move towards the target. A PID controller was used to determine appropriate vehicle speed for maintaining a certain distance from the target. Field tests were conducted to evaluate robustness and performance of the system. Results showed that the robot vehicle was correctly and effectively guided to follow the target while preventing collisions with disturbances. RMS errors were 1.0m and 1.3° in terms of space and deflection angle, respectively, when following a target that moved in a straight line at an average speed of 1.1m/s.