Visual Tracking Research Articles

A novel method for measurements of sleep/wake states, feeding and drinking behaviors using the tracking technique of 3D positions in freely moving mice

We have previously developed a 3D video tracking system which enables us to analyze long-term quantitative analysis of gene expression in freely moving mice. In the present study, we improved 3D video tracking and developed a system that analyzes more detailed behavioral data. We succeeded in simultaneously analyzing sleep-wake, feeding, and drinking behavior rhythms in the same individual using our tracking system. This system will make it possible to measure gene expression in each tissue in vivo in real time in relation to the various behavioral rhythms mentioned above.

Review and Analysis of RGBT Single Object Tracking Methods: A Fusion Perspective

Visual tracking is a fundamental task in computer vision with significant practical applications in various domains, including surveillance, security, robotics, and human-computer interaction. However, it may face limitations in visible light data, such as low-light environments, occlusion, and camouflage, which can significantly reduce its accuracy. To cope with these challenges, researchers have explored the potential of combining the visible and infrared modalities to improve tracking performance. By leveraging the complementary strengths of visible and infrared data, RGB-infrared fusion tracking has emerged as a promising approach to address these limitations and improve tracking accuracy in challenging scenarios. In this article, we present a review on RGB-infrared fusion tracking. Specifically, we categorize existing RGBT tracking methods into four categories based on their underlying architectures, feature representations, and fusion strategies, namely feature decoupling based method, feature selecting based method, collaborative graph tracking method, and traditional fusion method. Furthermore, we provide a critical analysis of their strengths, limitations, representative methods, and future research directions. To further demonstrate the advantages and disadvantages of these methods, we present a review of publicly available RGBT tracking datasets and analyze the main results on public datasets. Moreover, we discuss some limitations in RGBT tracking at present and provide some opportunities and future directions for RGBT visual tracking, such as dataset diversity, unsupervised and weakly supervised applications. In conclusion, our survey aims to serve as a useful resource for researchers and practitioners interested in the emerging field of RGBT tracking, and to promote further progress and innovation in this area.

Efficient Vibration Measurement and Modal Shape Visualization Based on Dynamic Deviations of Structural Edge Profiles.

As a non-contact method, vision-based measurement for vibration extraction and modal parameter identification has attracted much attention. In most cases, artificial textures are crucial elements for visual tracking, and this feature limits the application of vision-based vibration measurement on textureless targets. As a computation technique for visualizing subtle variations in videos, the video magnification technique can analyze modal responses and visualize modal shapes, but the efficiency is low, and the processing results contain clipping artifacts. This paper proposes a novel method for the application of a modal test. In contrast to the deviation magnification that exaggerates subtle geometric deviations from only a single image, the proposed method extracts vibration signals with sub-pixel accuracy on edge positions by changing the perspective of deviations from space to timeline. Then, modal shapes are visualized by decoupling all spatial vibrations following the vibration theory of continuous linear systems. Without relying on artificial textures and motion magnification, the proposed method achieves high operating efficiency and avoids clipping artifacts. Finally, the effectiveness and practical value of the proposed method are validated by two laboratory experiments on a cantilever beam and an arch dam model.

An optically responsive cancer vaccine for inducing robust anti-tumor immunity by apoptotic body carrying nanoadjuvants

A cancer vaccine based on nano-adjuvant fusion represents an effective strategy for anti-tumor immunity. Compared with other cancer vaccine formulations, the ones based on apoptotic bodies not only possess higher load-bearing capacity and biosafety, but also present greater clinical translational potential. In this study, we developed an optically responsive cancer vaccine that induced robust anti-tumor immunity by encapsulating genipin-crosslinked nano-adjuvants in tumor cell apoptotic bodies. The vaccine may potentially be used for prophylactic or therapeutic purposes. As an antigen source and a cargo vehicle, apoptotic bodies can co-deliver antigens and adjuvants simultaneously to antigen-presenting cells to activate T cells. Nano-adjuvants promote the targeted delivery of adjuvants for effective immune responses. Genipin-crosslinked nano-adjuvants enable visual tracking of the tumor vaccine. In addition, the photothermal properties of genipin-crosslinked nano-adjuvants promote lysosomal escape of tumor antigens to enhance cross-presentation. These characteristics enable visualization and controllability of cancer vaccines, thereby avoiding the need for exogenous photosensitive reagents. Apoptotic body-based vaccines can be used to prevent or treat tumors by triggering a strong systemic immune response by promoting T-cell infiltration and improving the tumor immune microenvironment.

Measuring activity-rest rhythms under different acclimation periods in a marine fish using automatic deep learning-based video tracking

ABSTRACT Most organisms synchronize to an approximately 24-hour (circadian) rhythm. This study introduces a novel deep learning-powered video tracking method to assess the stability, fragmentation, robustness and synchronization of activity rhythms in Xyrichtys novacula. Experimental X. novacula were distributed into three groups and monitored for synchronization to a 14/10 hours of light/dark to assess acclimation to laboratory conditions. Group GP7 acclimated for 1 week and was tested from days 7 to 14, GP14 acclimated for 14 days and was tested from days 14 to 21 and GP21 acclimated for 21 days and was tested from days 21 to 28. Telemetry data from individuals in the wild depicted their natural behavior. Wild fish displayed a robust and minimally fragmented rhythm, entrained to the natural photoperiod. Under laboratory conditions, differences in activity levels were observed between light and dark phases. However, no differences were observed in activity rhythm metrics among laboratory groups related to acclimation period. Notably, longer acclimation (GP14 and GP21) led to a larger proportion of individuals displaying rhythm synchronization with the imposed photoperiod. Our work introduces a novel approach for monitoring biological rhythms in laboratory conditions, employing a specifically engineered video tracking system based on deep learning, adaptable for other species.

Correlation filter based single object tracking: A review

In recent years, correlation filter-based (CF) tracking algorithms have gained momentum in the field of visual tracking. CF tracking algorithms have achieved compelling performance by addressing its limitations such as boundary effect and filter corruption during various tracking and target appearance variations. Many researchers have attempted to provide better efficiency and tracking results by extracting handcrafted and deep features either from vision sensors or specialized sensors in the CF tracking framework. Handcrafted features are integrated with deep features, thermal features, and depth features to prevent tracking failures during dense tracking challenges. To provide a detailed understanding of CF-based trackers, the tracking algorithms are categorized either as kernelized CF trackers or fusion-based CF trackers in this work. This is the first review of its kind, which categorizes various correlation-based tracking algorithms based on the key methodologies and the exploited features in the appearance model. Under each category, salient features, detailed overview, and current advancement are discussed and tabulated to provide future research directions. In addition, the performance of CF-based state-of-the-art is experimentally evaluated on multiple datasets namely, OTB100 and VOT2017 under tough tracking situations.

Decoder-ROI based Versatile Video Coding for Multi-Object Tracking Vision Task

The video encoding standards High Efficiency Video Coding (HEVC) and, more recently, Versatile Video Coding (VVC) have introduced significant advancements in multimedia communication applications, such as video conferencing, broadcasting, and notably, E-learning. However, recent developments in artificial intelligence (AI) and big data have given rise to an urgent need for a specialized video encoding model designed specifically for image and video analysis applications using machine vision. In this paper, we propose a novel video encoding approach that effectively combines the ROI Coding algorithm and the VVC encoding model. The proposed method identifies regions of interest within video frames through fundamental and deep features. Based on this, we propose an adaptive compression method for each frame block, ensuring both the execution performance of machine learning applications and minimal data encoding requirements. To achieve new coding scheme without adding bitrate, New feature extraction approach are utilizing only decoded information (Decoder-ROI). The results demonstrate that the Decoder-ROI achieved significant compression rate improvement when compared to standard and relevant VCM schemes. Furthermore, ROI exploitation contributes to a 3.25\% reduction in encoding time compared to the baseline VVC encoding standard.

LSOTB-TIR: A Large-Scale High-Diversity Thermal Infrared Single Object Tracking Benchmark.

Unlike visual object tracking, thermal infrared (TIR) object tracking methods can track the target of interest in poor visibility such as rain, snow, and fog, or even in total darkness. This feature brings a wide range of application prospects for TIR object-tracking methods. However, this field lacks a unified and large-scale training and evaluation benchmark, which has severely hindered its development. To this end, we present a large-scale and high-diversity unified TIR single object tracking benchmark, called LSOTB-TIR, which consists of a tracking evaluation dataset and a general training dataset with a total of 1416 TIR sequences and more than 643 K frames. We annotate the bounding box of objects in every frame of all sequences and generate over 770 K bounding boxes in total. To the best of our knowledge, LSOTB-TIR is the largest and most diverse TIR object tracking benchmark to date. We spilt the evaluation dataset into a short-term tracking subset and a long-term tracking subset to evaluate trackers using different paradigms. What's more, to evaluate a tracker on different attributes, we also define four scenario attributes and 12 challenge attributes in the short-term tracking evaluation subset. By releasing LSOTB-TIR, we encourage the community to develop deep learning-based TIR trackers and evaluate them fairly and comprehensively. We evaluate and analyze 40 trackers on LSOTB-TIR to provide a series of baselines and give some insights and future research directions in TIR object tracking. Furthermore, we retrain several representative deep trackers on LSOTB-TIR, and their results demonstrate that the proposed training dataset significantly improves the performance of deep TIR trackers. Codes and dataset are available at https://github.com/QiaoLiuHit/LSOTB-TIR.

A study on dual quaternion based cooperative relative navigation of multiple UAVs with monocular vision-inertial integration

This paper addresses a cooperative relative navigation problem for multiple aerial agents, relying on visual tracking information between vehicles. The research aims to investigate a sensor fusion architecture and algorithm that leverages partially available absolute navigation knowledge while exploiting collaborative visual interaction between vehicles in mission flight areas, where satellite navigation-denied regions are irregularly located. To achieve this, the paper introduces a new approach to defining the relative poses of cameras and develops a corresponding process to secure the relative pose information. This contrasts with previous research, which simply linearized the relative pose information of aircraft cameras into navigation states defined in an absolute coordinate system. Specifically, the target pose in relative navigation is defined, and the pose of the camera and feature points are directly derived using dual quaternion representation, which compactly represents both translation and rotation. Furthermore, a mathematical model for the relative pose of the camera is derived through the dual quaternion framework, enabling an explicit pose formulation of relative navigation. The study investigates navigation performance in typical mission flight scenarios using an in-house high-fidelity simulator and quantitatively highlights the contributions of the proposed scheme by comparing the navigation error performance. Consequently, the proposed method demonstrates to have navigation accuracy in decimeter level even in GNSS-denied environments and an improved 3D Root Mean Square (RMS) error by 30% smaller than the conventional absolute navigation framework.

Probabilistic Assignment With Decoupled IoU Prediction for Visual Tracking

Probabilistic Assignment With Decoupled IoU Prediction for Visual Tracking

Self-Supervised Tracking via Target-Aware Data Synthesis.

While deep-learning-based tracking methods have achieved substantial progress, they entail large-scale and high-quality annotated data for sufficient training. To eliminate expensive and exhaustive annotation, we study self-supervised (SS) learning for visual tracking. In this work, we develop the crop-transform-paste operation, which is able to synthesize sufficient training data by simulating various appearance variations during tracking, including appearance variations of objects and background interference. Since the target state is known in all synthesized data, existing deep trackers can be trained in routine ways using the synthesized data without human annotation. The proposed target-aware data-synthesis method adapts existing tracking approaches within a SS learning framework without algorithmic changes. Thus, the proposed SS learning mechanism can be seamlessly integrated into existing tracking frameworks to perform training. Extensive experiments show that our method: 1) achieves favorable performance against supervised (Su) learning schemes under the cases with limited annotations; 2) helps deal with various tracking challenges such as object deformation, occlusion (OCC), or background clutter (BC) due to its manipulability; 3) performs favorably against the state-of-the-art unsupervised tracking methods; and 4) boosts the performance of various state-of-the-art Su learning frameworks, including SiamRPN++, DiMP, and TransT.

Online Multi-Scale Classification and Global Feature Modulation for Robust Visual Tracking

Online Multi-Scale Classification and Global Feature Modulation for Robust Visual Tracking

ArchIMEDES: Computer Vision Tracking of the Inherent Changes to Structural Stability of Masonry Arch Bridges Resulting from Increased Bed Scour.

Scour-induced failure of masonry arch bridges is a common cause of bridge collapse, especially during extreme flood events. Over recent years there has been a significant increase in the numbers of documented scour induced failures of this bridge stock across Europe and the USA. The extent of the scour can erode the supporting soil or sediment around the bridge piers or abutments, leading to a loss of support for the structure. As the foundation becomes unstable, it can result in settlement or tilting of the bridge, potentially causing structural damage, while the ongoing nature of scour can result in the progressive deterioration of the bridge’s ability to safely transfer load. Traditional hydraulic adjustments, such as streamlining abutments with wing walls and using cutwaters on piers, do not significantly reduce scour depth for masonry arch bridges. This paper presents initial findings of an investigation that links the changes of response in load carrying capacity to the easement of support at the bridge foundation due to scour. Scale experiments on the FlexiArchTM under loading with realistic scour conditions is contributing to the development of an algorithm based on the relationship between the structural response of the FlexiArchTM and prevailing scour conditions. The computer vision-based monitoring package ArchIMEDES, developed as part of the research, will use this algorithm to predict current bridge scour conditions at any given time. This will allow for potential scour development to be identified and investigated before the bridge collapses, ensuring the security and safety of these vital pieces of infrastructure.

An efficient object tracking based on multi‐head cross‐attention transformer

AbstractObject tracking is an essential component of computer vision and plays a significant role in various practical applications. Recently, transformer‐based trackers have become the predominant method for tracking due to their robustness and efficiency. However, existing transformer‐based trackers typically focus solely on the template features, neglecting the interactions between the search features and the template features during the tracking process. To address this issue, this article introduces a multi‐head cross‐attention transformer for visual tracking (MCTT), which effectively enhance the interaction between the template branch and the search branch, enabling the tracker to prioritize discriminative feature. Additionally, an auxiliary segmentation mask head has been designed to produce a pixel‐level feature representation, enhancing and tracking accuracy by predicting a set of binary masks. Comprehensive experiments have been performed on benchmark datasets, such as LaSOT, GOT‐10k, UAV123 and TrackingNet using various advanced methods, demonstrating that our approach achieves promising tracking performance. MCTT achieves an AO score of 72.8 on the GOT‐10k.

Hunt-inspired Transformer for visual object tracking

This paper presents a hunt-inspired Transformer for visual object tracking, dubbed as HuntFormer. The HuntFormer focuses on robust target detection and identification, simulating natural hunting processes. Specifically, the HuntFormer comprises two essential module designs including a predictor for detection and a verifier for identification. The predictor emulates the detection stage by designing a motion trajectory guided particle filter, which identifies potential target locations by predicting the motion state within a particle filtering framework. The predictor utilizes spatio-temporal correlation scores between dynamic target templates and the search region to guide the learning process to generate a set of reliable particles. This enables the base tracker to narrow its search range to focus on the target, and swiftly re-detect the target in case of model drift. Once the target is re-detected, the verifier assesses the detection result as a reliable tracked item. The verifier initially maintains a dynamic memory that stores reliable target templates and their corresponding locations in the motion trajectory. It then models the uncertainty of appearance information within this memory probabilistically. The output uncertainty score determines whether the memory gets updated or not. Ultimately, the predictor and the verifier collaborate, ensuring a robust tracking outcome. Extensive evaluations on six challenging benchmark datasets demonstrate HuntFormer’s favorable performance against various state-of-the-art trackers. Notably, in the VOT-LT2022 tracking challenge, the HuntFormer won the third place with an F-score of 0.598, closely competing for the second place with an F-score of 0.600.

Reliable object tracking by multimodal hybrid feature extraction and transformer-based fusion

Visual object tracking, which is primarily based on visible light image sequences, encounters numerous challenges in complicated scenarios, such as low light conditions, high dynamic ranges, and background clutter. To address these challenges, incorporating the advantages of multiple visual modalities is a promising solution for achieving reliable object tracking. However, the existing approaches usually integrate multimodal inputs through adaptive local feature interactions, which cannot leverage the full potential of visual cues, thus resulting in insufficient feature modeling. In this study, we propose a novel multimodal hybrid tracker (MMHT) that utilizes frame-event-based data for reliable single object tracking. The MMHT model employs a hybrid backbone consisting of an artificial neural network (ANN) and a spiking neural network (SNN) to extract dominant features from different visual modalities and then uses a unified encoder to align the features across different domains. Moreover, we propose an enhanced transformer-based module to fuse multimodal features using attention mechanisms. With these methods, the MMHT model can effectively construct a multiscale and multidimensional visual feature space and achieve discriminative feature modeling. Extensive experiments demonstrate that the MMHT model exhibits competitive performance in comparison with that of other state-of-the-art methods. Overall, our results highlight the effectiveness of the MMHT model in terms of addressing the challenges faced in visual object tracking tasks.

The Effect of Repetitive Transcranial Magnetic Stimulation on Cognition in Diffuse Axonal Injury in a Rat Model.

Diffuse axonal injury (DAI) following sudden acceleration and deceleration can lead to cognitive function decline. Various treatments have been proposed. Repetitive transcranial magnetic stimulation (rTMS), a non-invasive stimulation technique, is a potential treatment for enhancing neuroplasticity in cases of brain injury. The therapeutic efficacy of rTMS on cognitive function remains unconfirmed. This study investigated the effects of rTMS and the underlying molecular biomechanisms using a rat model of DAI. Sprague-Dawley rats (n = 18) were randomly divided into two groups: one receiving rTMS after DAI and the other without brain stimulation. All rats were subjected to sudden acceleration and deceleration using a DAI modeling machine to induce damage. MRI was performed to confirm the DAI lesion. The experimental group received rTMS at a frequency of 1 Hz over the frontal cortex for 10 min daily for five days. To assess spatial memory, we conducted the Morris water maze (MWM) test one day post-brain damage and one day after the five-day intervention. A video tracking system recorded the escape latency. After post-MWM tests, all rats were euthanized, and their brain tissues, particularly from the hippocampus, were collected for immunohistochemistry and western blot analyses. The escape latency showed no difference on the MWM test after DAI, but a significant difference was observed after rTMS between the two groups. Immunohistochemistry and western blot analyses indicated increased expression of BDNF, VEGF, and MAP2 in the hippocampal brain tissue of the DAI-T group. In conclusion, rTMS improved cognitive function in the DAI rat model. The increased expression of BDNF, VEGF, and MAP2 in the DAI-T group supports the potential use of rTMS in treating cognitive impairments associated with DAI.

Material Testing for Physicists: Unraveling the Dissipative Nature of Silicone Elastomers via Ball Drop Testing

AbstractIsaac Newton once contemplated the fall of an apple, setting in motion a revolution in the understanding of gravity. In a similar spirit of curiosity and inquiry, here a journey is embarked upon to explore the intricate world of viscoelastic damping for polydimethylsiloxanes (PDMS). Inspired by the notion that even the simplest of phenomena can yield profound insights, a novel approach to study damping in silicone elastomers through a simple ball drop test is introduced. This novel solution allowes for precise measuring and analyzing the material's damping characteristics under various conditions. By carefully controlling the release and monitoring, the response of the falling ball by simple video tracking, valuable insights into the key viscoelastic properties of silicone blends are extracted, including rebound resilience, Young's modulus, and complex modulus. Through the analysis of trajectory data generated during the sphere's interaction with the silicone damper, dynamic and static material parameters are determined. Remarkably, these outcomes closely align with results obtained from cost‐intensive and high‐maintenance industrial measurement setups such as dynamic thermomechanical analysis (DTMA) or tensile testing. This approach not only simplifies the complexity of the system but also offers a cost‐effective and efficient means of gaining essential knowledge in material science.

Surface integrity of consecutive shot-peened additively manufactured 316L stainless steel

ABSTRACT The shot peening procedure enhances the mechanical characteristics with the transformation of compressive residual stresses which helps in the increment of operational lifespan of the product. The study comprehensively analyzed the effect of consecutive shot peening procedures up to four peening passes with the coverage increases 100% on each peening pass up to 400% on the laser powder bed fused austenitic stainless steel and compared with the as-built condition focusing on surface characteristics, mechanical characteristics, and corrosion potential. Surface damage of the peened samples was found very minimal with the absence of visible scan tracks and unmelted powder particles due to the controlled peening parameters. The inducement of compressive residual stresses (−625 MPa) was observed to be higher at the fourth consecutive peening passes without strain-induced martensite, with an increment of hardness proportion of 29.12% with a depth of 1 mm, and increment of corrosion resistance across each peening pass.

Multi-modal visual tracking based on textual generation

Multi-modal tracking has garnered significant attention due to its wide range of potential applications. Existing multi-modal tracking approaches typically merge data from different visual modalities on top of RGB tracking. However, focusing solely on the visual modality is insufficient due to the scarcity of tracking data. Inspired by the recent success of large models, this paper introduces a Multi-modal Visual Tracking Based on Textual Generation (MVTTG) approach to address the limitations of visual tracking, which lacks language information and overlooks semantic relationships between the target and the search area. To achieve this, we leverage large models to generate image descriptions, using these descriptions to provide complementary information about the target’s appearance and movement. Furthermore, to enhance the consistency between visual and language modalities, we employ prompt learning and design a Visual-Language Interaction Prompt Manager (V-L PM) to facilitate collaborative learning between visual and language domains. Experiments conducted with MVTTG on multiple benchmark datasets confirm the effectiveness and potential of incorporating image descriptions in multi-modal visual tracking.