Multimodal Tracking Research Articles

Multi-modal visual tracking based on textual generation

Multi-modal tracking has garnered significant attention due to its wide range of potential applications. Existing multi-modal tracking approaches typically merge data from different visual modalities on top of RGB tracking. However, focusing solely on the visual modality is insufficient due to the scarcity of tracking data. Inspired by the recent success of large models, this paper introduces a Multi-modal Visual Tracking Based on Textual Generation (MVTTG) approach to address the limitations of visual tracking, which lacks language information and overlooks semantic relationships between the target and the search area. To achieve this, we leverage large models to generate image descriptions, using these descriptions to provide complementary information about the target’s appearance and movement. Furthermore, to enhance the consistency between visual and language modalities, we employ prompt learning and design a Visual-Language Interaction Prompt Manager (V-L PM) to facilitate collaborative learning between visual and language domains. Experiments conducted with MVTTG on multiple benchmark datasets confirm the effectiveness and potential of incorporating image descriptions in multi-modal visual tracking.

RGBT tracking: A comprehensive review

In recent years, visual object tracking, as a prominent research area in computer vision, has garnered significant attention. To bolster the robustness of trackers across a spectrum of complex scenarios, researchers actively explore the synergistic potential of visible and thermal infrared images, aiming to design more potent tracking systems. This paper presents a comprehensive review of target tracking technology based on visible and thermal infrared information, encompassing three key aspects. Firstly, we categorize existing RGBT tracking methods into two main categories: traditional-based methods and deep learning-based methods. This classification facilitates a systematic understanding and comparison of the strengths and weaknesses of different approaches, providing a solid foundation for future research. Secondly, we focus on the evolution of RGBT datasets and analyze the performance of diverse tracking methods on these datasets. Research in this domain aids in evaluating the applicability of existing methods in real-world scenarios and offers guidance for future dataset construction. Finally, we delve into future research directions from multiple perspectives, including model design and dataset construction. In terms of model design, researchers are encouraged to explore more efficient feature extraction methods and innovative model fusion structures to further enhance tracker performance. Regarding dataset construction, increased attention should be given to ensure diversity in real-world scenarios, guaranteeing optimal tracker performance across a variety of complex conditions. In conclusion, this review makes a comprehensive analysis of the development of RGBT tracking from different perspectives, and provides a valuable reference for researchers in related fields such as multi-modal tracking and image fusion. By systematically classifying and analyzing existing research while outlining future research prospects, this review aims to foster the continued development of this field and inspire the emergence of more innovative work.

MINet: Modality interaction network for unified multi-modal tracking

While RGB-based trackers have made impressive progress, they still falter in complex scenarios, necessitating the exploration of multi-modal tracking strategies that leverage auxiliary modalities. However, most existing methods lack sufficient exploration and interaction of complementary information within and between modalities. To address this, we propose a Modality Interaction Network (MINet), a unified framework for multi-modal tracking. It consists of a Modality Representation Module (MRM) and a Memory Query Module (MQM). MRM enforces communications between different modalities by a designed Modality Interaction module (MIM) and fuses multi-modal information by a Modality Fuse Module (MFM) to generate more discriminative representation. MQM maintains historical multi-modal information and builds long-range dependencies between current and historical targets for tracking, which enhances the tracking performance, especially when targets undergo significant deformation and occlusions. To verify efficiency across different multi-modal tracking paradigms, we conduct extensive experiments, including RGB-D, RGB-T, and RGB-E. The experimental results demonstrate that in these multi-modal tracking tasks, the proposed MINet achieves outstanding performance compared to state-of-the-art trackers. Specifically, it outperforms them by 1% in RGB-D, 1.2% in RGB-T, and 1% in RGB-E tracking performance, respectively.

An automated methodology for whole-body, multimodality tracking of individual cancer lesions.

Objective. Manual analysis of individual cancer lesions to assess disease response is clinically impractical and requires automated lesion tracking methodologies. However, no methodology has been developed for whole-body individual lesion tracking, across an arbitrary number of scans, and acquired with various imaging modalities.Approach. This study introduces a lesion tracking methodology and benchmarked it using 2368Ga-DOTATATE PET/CT and PET/MR images of eight neuroendocrine tumor patients. The methodology consists of six steps: (1) alignment of multiple scans via image registration, (2) body-part labeling, (3) automatic lesion-wise dilation, (4) clustering of lesions based on local lesion shape metrics, (5) assignment of lesion tracks, and (6) output of a lesion graph. Registration performance was evaluated via landmark distance, lesion matching accuracy was evaluated between each image pair, and lesion tracking accuracy was evaluated via identical track ratio. Sensitivity studies were performed to evaluate the impact of lesion dilation (fixed versus automatic dilation), anatomic location, image modalities (inter- versus intra-modality), registration mode (direct versus indirect registration), and track size (number of time-points and lesions) on lesion matching and tracking performance.Main results. Manual contouring yielded 956 lesions, 1570 lesion-matching decisions, and 493 lesion tracks. The median residual registration error was 2.5 mm. The automatic lesion dilation led to 0.90 overall lesion matching accuracy, and an 88% identical track ratio. The methodology is robust regarding anatomic locations, image modalities, and registration modes. The number of scans had a moderate negative impact on the identical track ratio (94% for 2 scans, 91% for 3 scans, and 81% for 4 scans). The number of lesions substantially impacted the identical track ratio (93% for 2 nodes versus 54% for ≥5 nodes).Significance. The developed methodology resulted in high lesion-matching accuracy and enables automated lesion tracking in PET/CT and PET/MR.

Generative-Based Fusion Mechanism for Multi-Modal Tracking

Generative models (GMs) have received increasing research interest for their remarkable capacity to achieve comprehensive understanding. However, their potential application in the domain of multi-modal tracking has remained unexplored. In this context, we seek to uncover the potential of harnessing generative techniques to address the critical challenge, information fusion, in multi-modal tracking. In this paper, we delve into two prominent GM techniques, namely, Conditional Generative Adversarial Networks (CGANs) and Diffusion Models (DMs). Different from the standard fusion process where the features from each modality are directly fed into the fusion block, we combine these multi-modal features with random noise in the GM framework, effectively transforming the original training samples into harder instances. This design excels at extracting discriminative clues from the features, enhancing the ultimate tracking performance. Based on this, we conduct extensive experiments across two multi-modal tracking tasks, three baseline methods, and four challenging benchmarks. The experimental results demonstrate that the proposed generative-based fusion mechanism achieves state-of-the-art performance by setting new records on GTOT, LasHeR and RGBD1K. Code will be available at https://github.com/Zhangyong-Tang/GMMT.

Bi-directional Adapter for Multimodal Tracking

Due to the rapid development of computer vision, single-modal (RGB) object tracking has made significant progress in recent years. Considering the limitation of single imaging sensor, multi-modal images (RGB, infrared, etc.) are introduced to compensate for this deficiency for all-weather object tracking in complex environments. However, as acquiring sufficient multi-modal tracking data is hard while the dominant modality changes with the open environment, most existing techniques fail to extract multi-modal complementary information dynamically, yielding unsatisfactory tracking performance. To handle this problem, we propose a novel multi-modal visual prompt tracking model based on a universal bi-directional adapter, cross-prompting multiple modalities mutually. Our model consists of a universal bi-directional adapter and multiple modality-specific transformer encoder branches with sharing parameters. The encoders extract features of each modality separately by using a frozen, pre-trained foundation model. We develop a simple but effective light feature adapter to transfer modality-specific information from one modality to another, performing visual feature prompt fusion in an adaptive manner. With adding fewer (0.32M) trainable parameters, our model achieves superior tracking performance in comparison with both the full fine-tuning methods and the prompt learning-based methods. Our code is available: https://github.com/SparkTempest/BAT.

Robust RGB-T Tracking via Adaptive Modality Weight Correlation Filters and Cross-modality Learning

RGBT tracking is gaining popularity due to its ability to provide effective tracking results in a variety of weather conditions. However, feature specificity and complementarity have not been fully used in existing models that directly fuse the correlation filtering response, which leads to poor tracker performance. In this article, we propose correlation filters with adaptive modality weight and cross-modality learning (AWCM) ability to solve multimodality tracking tasks. First, we use weighted activation to fuse thermal infrared and visible modalities, and the fusion modality is used as an auxiliary modality to suppress noise and increase the learning ability of shared modal features. Second, we design modal weights through average peak-to-correlation energy coefficients to improve model reliability. Third, we propose consistency in using the fusion modality as an intermediate variable for joint learning consistency, thereby increasing tracker robustness via interactive cross-modal learning. Finally, we use the alternating direction method of multipliers algorithm to produce a closed solution and conduct extensive experiments on the RGBT234, VOT-TIR2019, and GTOT tracking benchmark datasets to demonstrate the superior performance of the proposed AWCM against compared to existing tracking algorithms. The code developed in this study is available at the following website. 1

Sparse mixed attention aggregation network for multimodal images fusion tracking

Recent years have witnessed the exciting performance of trackers based on Transformer. However, they usually separate the process of information extraction and integration, weakening the information interaction between the target and search region. In addition, they depend on traditional Transformer to model the long range dependency, which leads to a lack of focus on the primary information needed by high-accuracy trackers. In this paper, a sparse mixed attention aggregation model is proposed for robust tracking based on visible and thermal infrared images. To be specific, a backbone network composed of sparse mixed attention is designed to achieve information extraction and integration. This is helpful to obtain specific discriminative feature information and enhance their communication. To give full play to the complementary visible and thermal information, a confidence aware aggregation network is designed, which can learn the reliable confidence of visible and thermal branches. Finally, a corner-based localization head is introduced to estimate the target state. Extensive experiments on three large-scale multimodal tracking benchmarks demonstrate the superior tracking ability of the proposed tracker over other advanced trackers.

Exploring Opportunities for Multimodality and Multiple Devices in Food Journaling

Digital food journaling can support personal goals, such as weight loss and developing healthy eating behaviors. However, traditional manual tracking demands great effort, often leading to lapses or abandonment. We explore opportunities for journaling with multiple input modalities and devices, leveraging people's daily interactions with a range of technologies. We report on an extended analysis of 15 participants' experiences with ModEat, a prototype supporting journaling with several input modalities on phone, computer, and voice assistants. Participants' modality and device preferences were largely influenced by their goals, but they frequently deviated from those preferences depending on device availability, perceived affordances, and characteristics of foods eaten. Participants rarely combined input modalities in entries, but some described that doing so allowed for more detailed journaling or serve as a placeholder for later. We discuss advantages and drawbacks of multimodal tracking and potential strategies for improving interactions.

Multimodal Tracking of Theranostic Nanoparticles with CAR T Immunotherapy for Non-Hodgkin Lymphoma

HomeRadiology: Imaging CancerVol. 4, No. 4 PreviousNext Research HighlightsMultimodal Tracking of Theranostic Nanoparticles with CAR T Immunotherapy for Non-Hodgkin LymphomaKathleen D. McDaniel, Sophia Chakrabarty, Laycee Parsons, William M. MacCuaig, Lacey R. McNallyKathleen D. McDaniel, Sophia Chakrabarty, Laycee Parsons, William M. MacCuaig, Lacey R. McNallyKathleen D. McDanielSophia ChakrabartyLaycee ParsonsWilliam M. MacCuaigLacey R. McNallyPublished Online:Jul 22 2022https://doi.org/10.1148/rycan.229015MoreSectionsFull textPDF ToolsImage ViewerAdd to favoritesCiteTrack CitationsPermissionsReprints ShareShare onFacebookTwitterLinked In Highlighted ArticleShi B, Li D, Yao W, et al. Multifunctional theranostic nanoparticles for multi-modal imaging-guided CAR-T immunotherapy and chemo-photothermal combinational therapy of non-Hodgkin’s lymphoma. Biomater Sci 2022;10(10):2577–2589. doi: https://doi.org/10.1039/d1bm01982a Crossref, Medline, Google ScholarArticle HistoryPublished online: July 22 2022 FiguresReferencesRelatedDetailsRecommended Articles Spectrum of CT Findings in Thoracic Extranodal Non-Hodgkin LymphomaRadioGraphics2017Volume: 37Issue: 2pp. 439-461Non–Small Cell Lung Cancer with Resistance to EGFR-TKI Therapy: CT Characteristics of T790M Mutation–positive CancerRadiology2018Volume: 289Issue: 1pp. 227-237Imaging Primer on Chimeric Antigen Receptor T-Cell Therapy for RadiologistsRadioGraphics2022Volume: 42Issue: 1pp. 176-194Pancreatic Adenocarcinoma Staging in the Era of Preoperative Chemotherapy and Radiation TherapyRadiology2018Volume: 287Issue: 2pp. 374-390Updates in Diagnosis, Management, and Treatment of NeuroblastomaRadioGraphics2018Volume: 38Issue: 2pp. 566-580See More RSNA Education Exhibits Imaging Lymphomas of Head and Neck: State of the ArtDigital Posters2020Microbiota-Induced Gastrointestinal Malignancies: A Comprehensive ReviewDigital Posters2019Abdominal Manifestations of Extranodal Lymphoma in Patients With or Without Immune-compromised Disease: CT Features and Differential DiagnosisDigital Posters2020 RSNA Case Collection Pancreatic lymphomaRSNA Case Collection2020Primary Calvarial Lymphoma RSNA Case Collection2021Bing-Neel syndromeRSNA Case Collection2021 Vol. 4, No. 4 Metrics Altmetric Score PDF download

Fault-tolerant adaptive PID switched control of robot manipulator based on average dwell time

The multi-mode tracking control of robot manipulators with unknown dynamics is investigated in this paper. In view of varying loads, robot manipulators are modeled as multi-mode switched systems. Adaptive PID (proportion–integration–differentiation)–like switched controllers are designed via the error transformation and multiple Lyapunov function method, which can make the error system practical stability within prescribed time under a class of switching signals satisfying average dwell time. Furthermore, two cases under the healthy actuator and faulty actuator are discussed. For the case of the partial failure of the actuator, the fault-tolerant adaptive PID-like controller is presented, which consists of two parts. Finally, a simulation example is given to demonstrate the effectiveness of the proposed method.

Multi-Modal Sparse Tracking by Jointing Timing and Modal Consistency

In this paper, we propose a multi-modal sparse tracking by jointing timing and modal consistency to locate the target location with the similarity of multiple local appearances. First, we propose an alignable patching strategy for red-green-blue (RGB) color mode and thermal infrared mode to adapt to the local changes of the target. Second, we propose a consistency expression of the corresponding aligned patches between the modes and the correlation of the gaussian mapping within mode to reconstruct the target judgment likelihood function. Finally, we propose an updating scenario based on timing correlation and mode sparsity to fit with the target changes. According to the experimental results, significant improvement in terms of tracking accuracy can be achieved on average compared with the state-of-the-art algorithms. The source code of our algorithm is available on https://github.com/Liincq/tracker .

Mesenchymal Stem Cells Delivery in Individuals with Different Pathologies: Multimodal Tracking, Safety and Future Applications

Due to their ease of isolation and their properties, mesenchymal stem cells (MSCs) have been widely investigated. MSCs have been proved capable of migration towards areas of inflammation, including tumors. Therefore, they have been suggested as vectors to carry therapies, specifically to neoplasias. As most of the individuals joining clinical trials that use MSCs for cancer and other pathologies are carefully recruited and do not suffer from other diseases, here we decided to study the safety and application of iv-injected MSCs in animals simultaneously induced with different inflammatory pathologies (diabetes, wound healing and tumors). We studied this by in vitro and in vivo approaches using different gene reporters (GFP, hNIS, and f-Luc) and non-invasive techniques (PET, BLI, or fluorescence). Our results found that MSCs reached different organs depending on the previously induced pathology. Moreover, we evaluated the property of MSCs to target tumors as vectors to deliver adenoviruses, including the interaction between tumor microenvironment and MSCs on their arrival. Mechanisms such as transdifferentiation, MSC fusion with cells, or paracrine processes after MSCs homing were studied, increasing the knowledge and safety of this new therapy for cancer.

Aims and criteria for advancing technology management research at the Journal of Operations Management

Aims and criteria for advancing technology management research at the <scp>Journal of Operations Management</scp>

Correlation Filters Based on Strong Spatio-Temporal for Robust RGB-T Tracking

In this paper, we propose a strong spatio-temporal mechanism with correlation filters to solve multi-modality tracking tasks. First, we use the features of the previous four frames as spatio-temporal features, then aggregate the spatio-temporal features into the filters learning and positioning of the adjacent frame. Second, we enhance the temporal and spatial characteristics of the current frame filter by learning the previous four frame filters and spatial penalty. From the experimental results on the GTOT, VOT-TIR2019 and RGBT234 datasets, our strong spatio-temporal correlation filters has achieved excellent performance.

Multimodal Tracking of Hematopoietic Stem Cells from Young and Old Mice Labeled with Magnetic–Fluorescent Nanoparticles and Their Grafting by Bioluminescence in a Bone Marrow Transplant Model

This study proposes an innovative way to evaluate the homing and tracking of hematopoietic stem cells from young and old mice labeled with SPIONNIRF-Rh conjugated with two types of fluorophores (NIRF and Rhodamine), and their grafting by bioluminescence (BLI) in a bone marrow transplant (BMT) model. In an in vitro study, we isolated bone marrow mononuclear cells (BM-MNC) from young and old mice, and analyzed the physical–chemical characteristics of SPIONNIRF-Rh, their internalization, cell viability, and the iron quantification by NIRF, ICP-MS, and MRI. The in vivo study was performed in a BMT model to evaluate the homing, tracking, and grafting of young and old BM-MNC labeled with SPIONNIRF-Rh by NIRF and BLI, as well as the hematological reconstitution for 120 days. 5FU influenced the number of cells isolated mainly in young cells. SPIONNIRF-Rh had adequate characteristics for efficient internalization into BM-MNC. The iron load quantification by NIRF, ICP-MS, and MRI was in the order of 104 SPIONNIRF-Rh/BM-MNC. In the in vivo study, the acute NIRF evaluation showed higher signal intensity in the spinal cord and abdominal region, and the BLI evaluation allowed follow-up (11–120 days), achieving a peak of intensity at 30 days, which remained stable around 108 photons/s until the end. The hematologic evaluation showed similar behavior until 30 days and the histological results confirm that iron is present in almost all tissue evaluated. Our results on BM-MNC homing and tracking in the BMT model did not show a difference in migration or grafting of cells from young or old mice, with the hemogram analysis trending to differentiation towards the myeloid lineage in mice that received cells from old animals. The cell homing by NIRF and long term cell follow-up by BLI highlighted the relevance of the multimodal nanoparticles and combined techniques for evaluation.

The degradation regulation of 3D printed scaffolds for promotion of osteogenesis and in vivo tracking

Scaffold degradation regulation has become the key aspects in promoting bone regeneration and reconstruction. In the study, a novel stepwise-degraded PLGA/PCL/HA:Yb/Ho/Zn (PPHZ) scaffold was designed to compare with the fast-degradable PLGA/HA:Yb/Ho/Zn (PLHZ) scaffold to investigate the effects of scaffold degradation regulation on osteogenesis and to trace the scaffold degradation process. Micro-CT reconstruction, confocal fluorescence imaging, and histological sections were used for the analyses of the harvested animal samples, which demonstrated that the degradation regulated stepwise-degraded PPHZ scaffold maintaining higher mechanical strength and longer scaffold integrity is more advantageous for in vivo osteogenesis than the fast-degradable PLHZ scaffold. Meanwhile, the multifunctional HZ particles would help the anti-infection and tracking the distribution during scaffold degradation. The PPHZ scaffold may benefit future repair of anti-infective bone defects and in vivo multimodal tracking.

Tuning the Surface Chemistry of Second-Harmonic-Active Lithium Niobate Nanoprobes Using a Silanol-Alcohol Condensation Reaction.

The surface functionalization of nanoparticles (NPs) is of great interest for improving the use of NPs in, for example, therapeutic and diagnostic applications. The conjugation of specific molecules with NPs through the formation of covalent linkages is often sought to provide a high degree of colloidal stability and biocompatibility, as well as to provide functional groups for further surface modification. NPs of lithium niobate (LiNbO3) have been explored for use in second-harmonic-generation (SHG)-based bioimaging, expanding the applications of SHG-based microscopy techniques. The efficient use of SHG-active LiNbO3 NPs as probes will, however, require the functionalization of their surfaces with molecular reagents such as polyethylene glycol and fluorescent molecules to enhance their colloidal and chemical stability and to enable a correlative imaging platform. Herein, we demonstrate the surface functionalization of LiNbO3 NPs through the covalent attachment of alcohol-based reagents through a silanol-alcohol condensation reaction. Alcohol-based reagents are widely available and can have a range of terminal functional groups such as carboxylic acids, amines, and aldehydes. Attaching these molecules to NPs through the silanol-alcohol condensation reaction could diversify the reagents available to modify NPs, but this reaction pathway must first be established as a viable route to modifying NPs. This study focuses on the attachment of a linear alcohol functionalized with carboxylic acid and its use as a reactive group to further tune the surface chemistry of LiNbO3 NPs. These carboxylic acid groups were reacted to covalently attach other molecules to the NPs using copper-free click chemistry. This derivatization of the NPs provided a means to covalently attach polyethylene glycols and fluorescent probes to the NPs, reducing NP aggregation and enabling multimodal tracking of SHG nanoprobes, respectively. This extension of the silanol-alcohol condensation reaction to functionalize the surfaces of LiNbO3 NPs can be extended to other types of nanoprobes for use in bioimaging, biosensing, and photodynamic therapies.

Visual Object Multimodality Tracking Based on Correlation Filters for Edge Computing

In recent years, visual object tracking has become a very active research field which is mainly divided into the correlation filter-based tracking and deep learning (e.g., deep convolutional neural network and Siamese neural network) based tracking. For target tracking algorithms based on deep learning, a large amount of computation is required, usually deployed on expensive graphics cards. However, for the rich monitoring devices in the Internet of Things, it is difficult to capture all the moving targets in each device in real time, so it is necessary to perform hierarchical processing and use tracking based on correlation filtering in insensitive areas to alleviate the local computing pressure. In sensitive areas, upload the video stream to a cloud computing platform with a faster computing speed to perform an algorithm based on deep features. In this paper, we mainly focus on the correlation filter-based tracking. In the correlation filter-based tracking, the discriminative scale space tracker (DSST) is one of the most popular and typical ones which is successfully applied to many application fields. However, there are still some improvements that need to be further studied for DSST. One is that the algorithms do not consider the target rotation on purpose. The other is that it is a very heavy computational load to extract the histogram of oriented gradient (HOG) features from too many patches centered at the target position in order to ensure the scale estimation accuracy. To address these two problems, we introduce the alterable patch number for target scale tracking and the space searching for target rotation tracking into the standard DSST tracking method and propose a visual object multimodality tracker based on correlation filters (MTCF) to simultaneously cope with translation, scale, and rotation in plane for the tracked target and to obtain the target information of position, scale, and attitude angle at the same time. Finally, in Visual Tracker Benchmark data set, the experiments are performed on the proposed algorithms to show their effectiveness in multimodality tracking.

Multimodal Multiobject Tracking by Fusing Deep Appearance Features and Motion Information

Multiobject Tracking (MOT) is one of the most important abilities of autonomous driving systems. However, most of the existing MOT methods only use a single sensor, such as a camera, which has the problem of insufficient reliability. In this paper, we propose a novel Multiobject Tracking method by fusing deep appearance features and motion information of objects. In this method, the locations of objects are first determined based on a 2D object detector and a 3D object detector. We use the Nonmaximum Suppression (NMS) algorithm to combine the detection results of the two detectors to ensure the detection accuracy in complex scenes. After that, we use Convolutional Neural Network (CNN) to learn the deep appearance features of objects and employ Kalman Filter to obtain the motion information of objects. Finally, the MOT task is achieved by associating the motion information and deep appearance features. A successful match indicates that the object was tracked successfully. A set of experiments on the KITTI Tracking Benchmark shows that the proposed MOT method can effectively perform the MOT task. The Multiobject Tracking Accuracy (MOTA) is up to 76.40% and the Multiobject Tracking Precision (MOTP) is up to 83.50%.