Visual Tracking Research Articles

Visual tracking based on spatiotemporal transformer and fusion sequences

Currently, Transformer-based visual tracking methods have exhibited impressive performance. However, despite their widespread adoption, they still have certain limitations. For example, the design of the Transformer framework is somewhat original and redundant, resulting in lower efficiency. In addition, their application methods lack time consideration, and there is a lack of spatiotemporal correlation between tracking video sequences and predicting coordinate sequences, making it difficult to effectively integrate, and the robustness of corresponding tracking templates is insufficient. To address these issues, we propose a new visual tracking method (STFS). Firstly, it introduces a novel Flatten Transformer architecture, which, in comparison to previous modules, offers enhanced efficiency and expressiveness. Secondly, it takes multi frame feature maps and bounding box coordinates as inputs, integrates spatiotemporal information through the spatiotemporal sequence attention module, and provides relevant sequences for historical trend prediction. Finally, it uses diffusion methods to construct tracking templates and improve stability. To verify the performance of the tracker, we conducted experiments on benchmark datasets including GOT-10 K, LaSOT, TrackingNet, VOT2020, OTB100, and UAV123. The results demonstrate that STFS has achieved competitive experimental results.

Block based motion estimation model using CNN with representative point matching algorithm for object tracking in videos

Motion estimation is considered significant for tracking the movement of an object in video sequences, and it is widely used in various video processing applications. Traditionally, many researchers focus on pixel-based motion estimation for object tracking, but it experienced increased computation time and cost. To reduce computation time, the utilization of a block-based motion estimation approach for object tracking is a recent trend. The existing block-based approach faces difficulty in finding representative points within the intensity domain. Therefore, this current research merged the deep learning approach with a block-matching algorithm for achieving efficient object tracking. In this proposed work, initially, video sequences are collected from a benchmark video dataset. Then, the acquired video sequences are segmented into frames. From the segmented frames, current and previous frames are considered for motion estimation. Frames are sent for the data augmentation process in which the process of flipping, cropping, and rotation is carried out. Then, the augmented frames are sent into Convolutional Neural Network (CNN) for feature extraction. Representative Point Matching (RPM) is used to estimate the motion vector based on the extracted features. After estimating the motion vector, the similarity between two consecutive frames is found using Structural Similarity Index (SSIM) technique. Finally, based on the similarity score, the movement of an object in the video is tracked effectively. Simulation analysis of the proposed block-based motion estimation model is done by evaluating some performance metrics. RMSE, PSNR, Execution Time, SSIM, and accuracy obtained for the proposed model are 27.5, 26.5 db, 31 sec, 0.91, and 94 %. This analysis suggested that the proposed CNN-RPM motion estimation model performs better in tracking the movement of the object.

Multi-modal interaction with token division strategy for RGB-T tracking

RGB-T tracking takes visible and infrared images as inputs, which is an extended application of multi-modal fusion in the field of visual object tracking. The complementarity between visible and infrared modalities can enhance the robustness of tracker in complex scenes. Cross-modal interaction can facilitate the fusion and synergy of different modalities, but most previous methods lack clear target information in multi-modal fusion, leading to some undesired cross-relation in interaction. To reduce these undesired cross-relations, we propose a Multi-modal Interaction scheme Guided by Token Division strategy (MIGTD). This scheme divides the input multi-modal tokens into several categories and restricts the interaction between tokens by setting different rules. The above operation is implemented in parallel through an attention masking strategy. To accurately classify search tokens, an instance segmentation task with box-supervised loss is employed. We conduct extensive experiments on three popular benchmark datasets, RGBT234, LasHeR and VTUAV. The experimental results indicate that the tracker proposed in this article reach the world’s advanced level in performance.

Gaussian-based adaptive frame skipping for visual object tracking

Gaussian-based adaptive frame skipping for visual object tracking

Magnetic-Controlled Microrobot: Real-Time Detection and Tracking through Deep Learning Approaches.

As one of the most significant research topics in robotics, microrobots hold great promise in biomedicine for applications such as targeted diagnosis, targeted drug delivery, and minimally invasive treatment. This paper proposes an enhanced YOLOv5 (You Only Look Once version 5) microrobot detection and tracking system (MDTS), incorporating a visual tracking algorithm to elevate the precision of small-target detection and tracking. The improved YOLOv5 network structure is used to take magnetic bodies with sizes of 3 mm and 1 mm and a magnetic microrobot with a length of 2 mm as the pretraining targets, and the training weight model is used to obtain the position information and motion information of the microrobot in real time. The experimental results show that the accuracy of the improved network model for magnetic bodies with a size of 3 mm is 95.81%, representing an increase of 2.1%; for magnetic bodies with a size of 1 mm, the accuracy is 91.03%, representing an increase of 1.33%; and for microrobots with a length of 2 mm, the accuracy is 91.7%, representing an increase of 1.5%. The combination of the improved YOLOv5 network model and the vision algorithm can effectively realize the real-time detection and tracking of magnetically controlled microrobots. Finally, 2D and 3D detection and tracking experiments relating to microrobots are designed to verify the robustness and effectiveness of the system, which provides strong support for the operation and control of microrobots in an in vivo environment.

The role of prediction and visual tracking strategies during manual interception: An exploration of individual differences.

The interception (or avoidance) of moving objects is a common component of various daily living tasks; however, it remains unclear whether precise alignment of foveal vision with a target is important for motor performance. Furthermore, there has also been little examination of individual differences in visual tracking strategy and the use of anticipatory gaze adjustments. We examined the importance of in-flight tracking and predictive visual behaviors using a virtual reality environment that required participants (n = 41) to intercept tennis balls projected from one of two possible locations. Here, we explored whether different tracking strategies spontaneously arose during the task, and which were most effective. Although indices of closer in-flight tracking (pursuit gain, tracking coherence, tracking lag, and saccades) were predictive of better interception performance, these relationships were rather weak. Anticipatory gaze shifts toward the correct release location of the ball provided no benefit for subsequent interception. Nonetheless, two interceptive strategies were evident: 1) early anticipation of the ball's onset location followed by attempts to closely track the ball in flight (i.e., predictive strategy); or 2) positioning gaze between possible onset locations and then using peripheral vision to locate the moving ball (i.e., a visual pivot strategy). Despite showing much poorer in-flight foveal tracking of the ball, participants adopting a visual pivot strategy performed slightly better in the task. Overall, these results indicate that precise alignment of the fovea with the target may not be critical for interception tasks, but that observers can adopt quite varied visual guidance approaches.

LSTM-QDCNN: long short-term memory and quantum dilated convolutional neural network enabled occlusion percentage prediction

ABSTRACT Occlusion detection is vital in longer video sequences of real-time scenarios, as deformable objects generally occlude themselves or be occluded by exterior objects eventually. Occlusion can affect accuracy of the entire model significantly, because optical flow may reveal random attributes in occluded as well as neighbourhood areas. Occlusion is a challenging issue in real-world visual object tracking. Several techniques learn imprecise appearance of target while it becomes occluded by another object in the scene. Occlusion is a complicated issue that causes target loss in tracking procedure. Here, long short-term memory fused quantum dilated convolutional neural network (LSTM-QDCNN) is presented for occlusion percentage prediction. Initially, video frames are extracted from considered input video. The extracted frame is pre-processed by adaptive fuzzy filtering, and object localisation is accomplished by employing sparse fuzzy c-means with local optimal oriented pattern feature. Thereafter, occlusion category detection is conducted by deep Q-network. Lastly, occlusion percentage prediction is done by utilising LSTM-QDCNN. Additionally, LSTM-QDCNN is designed by merging deep LSTM (DLSTM) with quantum dilated convolutional neural network (QDCNN). Furthermore LSTM-QDCNN achieved maximum multiple-object tracking precision of 0.892 and minimum tracking distance of 3.714. In addition, LSTM-QDCNN achieved tracking number about 18.

Visual Servoing Trajectory Tracking and Depth Identification for Mobile Robots With Velocity Saturation Constraints

Visual Servoing Trajectory Tracking and Depth Identification for Mobile Robots With Velocity Saturation Constraints

D-VAT: End-to-End Visual Active Tracking for Micro Aerial Vehicles

D-VAT: End-to-End Visual Active Tracking for Micro Aerial Vehicles

MixFormer: End-to-End Tracking With Iterative Mixed Attention.

Visual object tracking often employs a multi-stage pipeline of feature extraction, target information integration, and bounding box estimation. To simplify this pipeline and unify the process of feature extraction and target information integration, in this paper, we present a compact tracking framework, termed as MixFormer, built upon transformers. Our core design is to utilize the flexibility of attention operations, and we propose a Mixed Attention Module (MAM) for simultaneous feature extraction and target information integration. This synchronous modeling scheme allows us to extract target-specific discriminative features and perform extensive communication between target and search area. Based on MAM, we build our MixFormer trackers simply by stacking multiple MAMs and placing a localization head on top. Specifically, we instantiate two types of MixFormer trackers, a hierarchical tracker MixCvT, and a non-hierarchical simple tracker MixViT. For these two trackers, we investigate a series of pre-training methods and uncover the different behaviors between supervised pre-training and self-supervised pre-training in our MixFormer trackers. We also extend the masked autoencoder pre-training to our MixFormer trackers and design the new competitive TrackMAE pre-training technique. Finally, to handle multiple target templates during online tracking, we devise an asymmetric attention scheme in MAM to reduce computational cost, and propose an effective score prediction module to select high-quality templates. Our MixFormer trackers set a new state-of-the-art performance on seven tracking benchmarks, including LaSOT, TrackingNet, VOT2020, GOT-10 k, OTB100, TOTB and UAV123. In particular, our MixViT-L achieves AUC scores of 73.3% on LaSOT, 86.1% on TrackingNet and 82.8% on TOTB.

Framework for multiple person identification using YOLOv8 detector: a transfer learning approach

Video surveillance extensively uses person detection and tracking technology based on video. The majority of person detection and classification techniques currently in use encounter challenges in video sequences brought on by occlusion, ambient lighting, and variations in human facial position. This paper proposed an effective person identification and classification system based on deep learning, which comprises a you only look once at version 8 (YOLOv8) detection and classification model, to classify human faces in video sequences accurately. This work proposes a new staff-detection and classification (S-DEC) dataset for comprehensive performance evaluation. visual tracker benchmark (VTB) standard database is used for performance comparison with the proposed S-DEC dataset. The proposed technique achieved 98.67% precision accuracy. For the S-DEC dataset, the system gave 94.67% accuracy in identifying facial images from a video sequence of 38 people addressing the pose variation occlusion challenge. Earlier methods used to provide approximately 85% to 90% results taking more execution time. Many existing techniques were successful in detecting people only-identification of the detected person has been done in limited papers. The proposed method uses the cross-stage partial connections (CSPDarknet53) model, integrated with YOLOv8, to achieve faster results. The proposed framework took 35 minutes to train a deep learning model. A testing time of 2 minutes ensured that the proposed framework outplayed other existing methodologies and successfully identified extra information about the detected person.

Enhancing Elementary School Students’ Visual Perceptual Skills: A Comparative Study of Oculus Quest 2 Exergames and Conventional Activities

The purpose of this study was to investigate potential differences between two exercise training programs among elementary school students: one based on Oculus Quest 2 exergames and the other on conventional activities, focusing on the Visual Pursuit (VP) skills of visual tracking and selective attention. The study comprised two cohorts: the Oculus Quest (OQ) group and the Typical Activity (TA) group. Engaged a total of forty (n=40) students, specifically selected from the fifth and sixth grades of two public elementary schools situated in the northern region of Greece. The participants, aged 10 to 12 years, consisted of 17 boys and 23 girls. Evaluation of the children’s VP abilities occurred through a Visual Pursuit Test administered prior to the interventions (pre-test), after the intervention sessions (post-test), and one month after the interventions (1-month retention test). This program spanned a duration of 6 weeks, with sessions held twice per week, each session lasting 30 minutes. Two-way analyses of variance with repeated measures, were conducted to determine effect of training program groups (TA, OQ) and measurements (pre-test, post-test, 1-month retention test) across time on VP performance. Analysis of the data illustrated that the post-test and 1-month retention test VP scores were remarkably greater than pre-test VP scores in both groups. In conclusion, the findings indicate that both utilizing Oculus Quest 2 VR and conventional activities as interventions show promise in enhancing the VP abilities of elementary school children. This suggests that VR technology, alongside typical methods, could be a valuable tool in Physical Education settings for improving cognitive skills among young learners.

РОЗРОБКА ЕФЕКТИВНОГО ТА ТОЧНОГО МЕТОДУ ВІДСТЕЖЕННЯ ОБʼЄКТІВ НА ВІДЕО НА МОБІЛЬНИХ ПРИСТРОЯХ

One of the top research topics in computer vision area is visual object tracking. Main goal is to obtain the target object’s location in the first video frame of a given video sequence. The recent innovations of deep neural networks, specifically Siamese networks has significant impact on visual object tracking. In spite of high accuracy and high results in academic benchmarks, there are drawbacks in current state-of-the-art approaches in particular compute-intensive and large memory footprint that cannot satisfy the performance requirements of real-world applications. The aim of this paper is to design a new lightweight framework for resource-efficient and accurate visual object tracking. To add, a new tracker of efficiency benchmark and protocol were introduced. Efficiency is defined in terms of both energy consumption and execution speed on edge devices. New dual template representation for object model adaptation was developed. The first template, static, fixes the original visible appearance and thus prevents deviation and, as a result, failures caused by adaptation. The other is dynamic; the state reflects the current conditions of assembly and the appearance of its object. Unlike STARK, which incorporates additional timing information by introducing a separate estimation prediction head, we introduce parameter-free module similarity as a template update rule optimized from the latest network. We show that the learned convex combination of two patterns is effective for tracking on multiple tests. A lightweight tracker was proposed, which includes functions, dual representation of patterns and pixel-by-pixel merged blocks in its compact network. The resulting FEAR-XS tracker runs at 205 FPS on the iPhone 11, which is 4.2 times faster than LightTrack and 26.6 times faster than Ocean, with high accuracy on many tests – no state-of-the-art tracker is more accurate and faster than any FEAR tracker. In addition, the algorithm is highly energy efficient.

RGBT tracking: A comprehensive review

In recent years, visual object tracking, as a prominent research area in computer vision, has garnered significant attention. To bolster the robustness of trackers across a spectrum of complex scenarios, researchers actively explore the synergistic potential of visible and thermal infrared images, aiming to design more potent tracking systems. This paper presents a comprehensive review of target tracking technology based on visible and thermal infrared information, encompassing three key aspects. Firstly, we categorize existing RGBT tracking methods into two main categories: traditional-based methods and deep learning-based methods. This classification facilitates a systematic understanding and comparison of the strengths and weaknesses of different approaches, providing a solid foundation for future research. Secondly, we focus on the evolution of RGBT datasets and analyze the performance of diverse tracking methods on these datasets. Research in this domain aids in evaluating the applicability of existing methods in real-world scenarios and offers guidance for future dataset construction. Finally, we delve into future research directions from multiple perspectives, including model design and dataset construction. In terms of model design, researchers are encouraged to explore more efficient feature extraction methods and innovative model fusion structures to further enhance tracker performance. Regarding dataset construction, increased attention should be given to ensure diversity in real-world scenarios, guaranteeing optimal tracker performance across a variety of complex conditions. In conclusion, this review makes a comprehensive analysis of the development of RGBT tracking from different perspectives, and provides a valuable reference for researchers in related fields such as multi-modal tracking and image fusion. By systematically classifying and analyzing existing research while outlining future research prospects, this review aims to foster the continued development of this field and inspire the emergence of more innovative work.

Controlling the fold: proprioceptive feedback in a soft origami robot.

We demonstrate proprioceptive feedback control of a one degree of freedom soft, pneumatically actuated origami robot and an assembly of two robots into a two degree of freedom system. The base unit of the robot is a 41 mm long, 3-D printed Kresling-inspired structure with six sets of sidewall folds and one degree of freedom. Pneumatic actuation, provided by negative fluidic pressure, causes the robot to contract. Capacitive sensors patterned onto the robot provide position estimation and serve as input to a feedback controller. Using a finite element approach, the electrode shapes are optimized for sensitivity at larger (more obtuse) fold angles to improve control across the actuation range. We demonstrate stable position control through discrete-time proportional-integral-derivative (PID) control on a single unit Kresling robot via a series of static set points to 17 mm, dynamic set point stepping, and sinusoidal signal following, with error under 3 mm up to 10 mm contraction. We also demonstrate a two-unit Kresling robot with two degree of freedom extension and rotation control, which has error of 1.7 mm and 6.1°. This work contributes optimized capacitive electrode design and the demonstration of closed-loop feedback position control without visual tracking as an input. This approach to capacitance sensing and modeling constitutes a major step towards proprioceptive state estimation and feedback control in soft origami robotics.

Review: Multiple Objects Detection and Tracking using Deep Learning Approach

Multiple Object Tracking (MOT) is a crucial tool with diverse applications, such as object detection, object counting, and security systems. Precise identification and monitoring of numerous objects are essential in several computer vision uses, such as monitoring, self-driving cars, and computer-human communication. Very little has been done to address occlusion problems in order to enable the best moving object tracking with detection The tracking of visual objects is one of the most important components of computer vision. The process of tracking an object (or a group of objects) across time is called object tracking. Visual object tracking is used to identify or link target items over successive video frames. In this study, we analyze the tracking-by-detection strategy, which includes YOLO-based detection and SORT-based tracking. This work elucidates a general approach to tracking and recognizing many objects with an emphasis on accuracy improvement. We aim to revolutionize computer vision by applying Non-Maximum Suppression (NMS) and Intersection over Union (IoU) approaches, and by combining the state-of-the-art YOLO NAS algorithm with conventional tracking methods or an alternative version of the YOLO Algorithm for object identification. It is expected that our work will have a major impact on many different applications, enabling more precise and reliable object tracking and detection in difficult real-world scenarios. Keywords—Multiple Object Detection, Kalman Filters Multiple Object Tracking, DeepSORT, YOLO, IoU, NMS.

Anti-Occlusion Visual Tracking Algorithm for UAVs with Multi-Feature Adaptive Fusion

Most of the existing trackers based on discriminative correlation filters use only one feature or a simple linear fusion of multiple features for object tracking, and most of them lack a mechanism to handle occlusions. This leads to poor tracking performance in rapidly changing and easily occluded scenarios, especially on unmanned aerial vehicle (UAV) platforms. To address this issue, this paper proposes an anti-occlusion visual tracking algorithm for UAVs with multi-feature adaptive fusion named multi-feature adaptive fusion and anti-occlusion tracker (MAFAOT). It introduces a novel approach for implementing an adaptive fusion of multiple features. This method transforms the multi-feature fusion problem into a maximization issue by designing a tracking quality evaluation index. It successfully achieves an adaptive fusion of gradient direction histogram and color histogram feature responses. MAFAOT also introduces an anti-occlusion update pool strategy, enabling the tracker to adapt dynamically to various complex scenarios, including occlusion and motion blur. The experimental results on the OTB100 and UAV123 datasets confirm the significant advantages of MAFAOT in terms of precision and success rate compared to other correlation filter-based algorithms. The proposed methods further enhance the expressiveness of the features and effectively avoid the problem of tracker contamination caused by occlusion. Furthermore, this paper applies the proposed methods to the kernelized correlation filters (KCF) algorithm. On the OTB100 dataset, the improved KCF algorithm shows an improvement of 10.94% in precision and 11.11% in success rate. On the UAV123 dataset, it shows an improvement of 14.53% in precision and 16.62% in success rate, further verifying the effectiveness and versatility of the proposed methods.

A Long-Term Video Tracking Method for Group-Housed Pigs

Pig tracking provides strong support for refined management in pig farms. However, long and continuous multi-pig tracking is still extremely challenging due to occlusion, distortion, and motion blurring in real farming scenarios. This study proposes a long-term video tracking method for group-housed pigs based on improved StrongSORT, which can significantly improve the performance of pig tracking in production scenarios. In addition, this research constructs a 24 h pig tracking video dataset, providing a basis for exploring the effectiveness of long-term tracking algorithms. For object detection, a lightweight pig detection network, YOLO v7-tiny_Pig, improved based on YOLO v7-tiny, is proposed to reduce model parameters and improve detection speed. To address the target association problem, the trajectory management method of StrongSORT is optimized according to the characteristics of the pig tracking task to reduce the tracking identity (ID) switching and improve the stability of the algorithm. The experimental results show that YOLO v7-tiny_Pig ensures detection applicability while reducing parameters by 36.7% compared to YOLO v7-tiny and achieving an average video detection speed of 435 frames per second. In terms of pig tracking, Higher-Order Tracking Accuracy (HOTA), Multi-Object Tracking Accuracy (MOTP), and Identification F1 (IDF1) scores reach 83.16%, 97.6%, and 91.42%, respectively. Compared with the original StrongSORT algorithm, HOTA and IDF1 are improved by 6.19% and 10.89%, respectively, and Identity Switch (IDSW) is reduced by 69%. Our algorithm can achieve the continuous tracking of pigs in real scenarios for up to 24 h. This method provides technical support for non-contact pig automatic monitoring.

Optimizing intrinsic representation for tracking

Learning target representations dedicated to tracking tasks is a promising direction in visual object tracking. Current state-of-the-art approaches exploit pixel appearance reconstruction of auto-encoders for representation training. Such pixel ways are redundant for the high-level semantics of tracking and suffer from an optimization gap where better appearance reconstruction does not correspond to better tracking performance. This article explores the problem of reconstruction misalignment of target representations in Siamese network architectures. We propose an optimization framework for intrinsic representation that restores the high-level feature loss between the networks on the reconstruction side and the target side. This intrinsic representation is generated by feeding the masked image tokens to the target side branch. Unlike the pixel ways, our approach learns to reconstruct the semantic latent level of representation for tracking objects. Moreover, we investigate feeding strategies from the information bottleneck perspective for masked patches to increase the masking performance yet reduce the computational complexity. Additionally, robust propagation mechanisms for the weight parameters of two-sided networks are also explored. Comprehensive evaluations on seven tracking benchmarks demonstrate our effectiveness: GOT-10k, LaSOT, TrackingNet, NFS, UAV123, TNL2K, and OTB100. Compared to the previous best pixel tracker, we outperform it on all seven datasets, with a maximum absolute score of 5.9% on average overlap and a Graphics Processing Unit (GPU) speed of 109 frames per second.

Striatin knock out induces a gain of function of INa and impaired Ca2+ handling in mESC-derived cardiomyocytes.

Striatin (Strn) is a scaffold protein expressed in cardiomyocytes (CMs) and alteration of its expression are described in various cardiac diseases. However, the alteration underlying its pathogenicity have been poorly investigated. We studied the role(s) of cardiac Strn gene (STRN) by comparing the functional properties of CMs, generated from Strn-KO and isogenic WT mouse embryonic stem cell lines. The spontaneous beating rate of Strn-KO CMs was faster than WT cells, and this correlated with a larger fast INa conductance and no changes in If. Paced (2-8 Hz) Strn-KO CMs showed prolonged action potential (AP) duration in comparison with WT CMs and this was not associated with changes in ICaL and IKr. Motion video tracking analysis highlighted an altered contraction in Strn-KO CMs; this was associated with a global increase in intracellular Ca2+, caused by an enhanced late Na+ current density (INaL) and a reduced Na+/Ca2+ exchanger (NCX) activity and expression. Immunofluorescence analysis confirmed the higher Na+ channel expression and a more dynamic microtubule network in Strn-KO CMs than in WT. Indeed, incubation of Strn-KO CMs with the microtubule stabilizer taxol, induced a rescue (downregulation) of INa conductance toward WT levels. Loss of STRN alters CMs electrical and contractile profiles and affects cell functionality by a disarrangement of Strn-related multi-protein complexes. This leads to impaired microtubules dynamics and Na+ channels trafficking to the plasma membrane, causing a global Na+ and Ca2+ enhancement.