Visual Tracking Research Articles

Automated analysis of operative video in surgical training: scoping review.

There is increasing availability of operative video for use in surgical training. Emerging technologies can now assess video footage and automatically generate metrics that could be harnessed to improve the assessment of operative performance. However, a comprehensive understanding of which technology features are most impactful in surgical training is lacking. The aim of this scoping review was to explore the current use of automated video analytics in surgical training. PubMed, Scopus, the Web of Science, and the Cochrane database were searched, to 29 September 2023, following PRISMA extension for scoping reviews (PRISMA-ScR) guidelines. Search terms included 'trainee', 'video analytics', and 'education'. Articles were screened independently by two reviewers to identify studies that applied automated video analytics to trainee-performed operations. Data on the methods of analysis, metrics generated, and application to training were extracted. Of the 6736 articles screened, 13 studies were identified. Computer vision tracking was the common method of video analysis. Metrics were described for processes (for example movement of instruments), outcomes (for example intraoperative phase duration), and critical safety elements (for example critical view of safety in laparoscopic cholecystectomy). Automated metrics were able to differentiate between skill levels (for example consultant versus trainee) and correlated with traditional methods of assessment. There was a lack of longitudinal application to training and only one qualitative study reported the experience of trainees using automated video analytics. The performance metrics generated from automated video analysis are varied and encompass several domains. Validation of analysis techniques and the metrics generated are a priority for future research, after which evidence demonstrating the impact on training can be established.

Effects of the insecticide acetamiprid and the fungicide azoxystrobin on locomotion activity and mushroom bodies of solitary bee Centris analis

Pesticide use is a major factor contributing to the global decline in bee populations. Sublethal effects, such as behavior alterations, are neglected in pesticide regulation for pollinators. However, these effects can bring important information to understanding the impacts of pesticides on bees' daily activities. In this study, we aimed to investigate the effects of the insecticide acetamiprid (7 ng/μL) and the fungicide azoxystrobin (10 ng/μL) on the behavior of the Neotropical solitary bee Centris analis. Female and male bees were exposed to these chemicals continuously for 48 h, followed by an additional 48 h without contaminated food, totaling 96 h of observation. We used five experimental groups: control, solvent control, insecticide, fungicide, and pesticide mixture (insecticide + fungicide). Behavioral alterations based on locomotion and light response were assessed by video tracking at 48 (end of pesticide exposure) and 96 h (end of bioassay). In addition, after recording bees at 96 h, the individuals were anesthetized for brain collection and histological evaluation of mushroom bodies to evaluate if pesticides can damage their neurons and impair the cognitive processes and responses of bees to sensory stimuli. Bees exposed to acetamiprid and pesticide mixture showed lethargic movements and impaired locomotion at 48 h. Notably, these behavioral effects were no longer evident after the bees consumed uncontaminated food for an additional 48 h, totaling 96 h from the start of pesticide exposure. Only fungicide exposure did not result in any behavioral or brain histological changes. Therefore, our study showed that acetamiprid at an estimated residual concentration, despite being classified as having low toxicity for bees, can cause significant initial locomotion disruption in solitary bees. These findings highlight the importance of considering sublethal effects in environmental risk assessment.

UAV-based field watermelon detection and counting using YOLOv8s with image panorama stitching and overlap partitioning

Accurate watermelon yield estimation is crucial to the agricultural value chain, as it guides the allocation of agricultural resources as well as facilitates inventory and logistics planning. The conventional method of watermelon yield estimation relies heavily on manual labor, which is both time-consuming and labor-intensive. To address this, this work proposes an algorithmic pipeline that utilizes unmanned aerial vehicle (UAV) videos for detection and counting of watermelons. This pipeline uses You Only Look Once version 8 s (YOLOv8s) with panorama stitching and overlap partitioning, which facilitates the overall number estimation of watermelons in field. The watermelon detection model, based on YOLOv8s and obtained using transfer learning, achieved a detection accuracy of 99.20 %, demonstrating its potential for application in yield estimation. The panorama stitching and overlap partitioning based detection and counting method uses panoramic images as input and effectively mitigates the duplications compared with the video tracking based detection and counting method. The counting accuracy reached over 96.61 %, proving a promising application for yield estimation. The high accuracy demonstrates the feasibility of applying this method for overall yield estimation in large watermelon fields.

Meta-reinforcement learning for active visual tracking about space non-cooperative object

Meta-reinforcement learning for active visual tracking about space non-cooperative object

Comparison of friction coefficient of static and sliding determination methods: conventional, video tracking and iot-based

Comparison of friction coefficient of static and sliding determination methods: conventional, video tracking and iot-based

Assessing habitat selection in the prawn Macrobrachium rosenbergii using the model toxicant copper and colonization as a test endpoint: Does prior exposure determine biochemical and behavioral responses?

Habitat selection by aquatic organisms is dependent on the availability of adequate conditions to support life and the benefits that the habitat provides. Contaminated environments tend to be less attractive to organisms because reduced habitat quality leads to increased maintenance costs. Consequently, reduced colonization of such disturbed habitats is an expected response. However, colonization has been understudied as an ecotoxicological test endpoint, despite its proven ability to assess habitat selection by populations across various taxa. The aim of the present study was to investigate whether previous exposure to copper could alter the colonization behavior of the freshwater prawn Macrobrachium rosenbergii along a non-forced exposure gradient of interconnected copper-contaminated habitats (0 to 500 µg/L) due to physiological and behavioral impairments. To assess this, post-larvae of M. rosenbergii were pre-exposed to 0, 50, 250 and 500 µg/L copper for a maximum period of 48 h. The physiological status and motility of the organisms after pre-exposure to copper were evaluated using behavioral endpoints (swimming activity by video tracking) and biochemical biomarkers (biotransformation, oxidative stress and neurotoxicity). The results indicated that pre-exposure to copper (at concentrations of 0, 50 and 500 µg/L) significantly influenced the median colonization concentration (CC50), which decreased from 270 µg/L to 109 µg/L. None of the assessed swimming parameters (speed, motility rate, exploration rate, and total distance) were affected by the pre-exposure to copper (0, 50 and 250 µg/L). Biochemically, cholinesterase levels were only affected in the prawn population exposed to 250 µg/L of copper. The present study provides a better understanding of the relevance of colonization as an ecotoxicological endpoint for assessing the spatial distribution of populations, including both new inhabitants and previously exposed organisms, in recovering habitats.

SSTtrack: A unified hyperspectral video tracking framework via modeling spectral-spatial-temporal conditions

Hyperspectral video contains rich spectral, spatial, and temporal conditions that are crucial for capturing complex object variations and overcoming the inherent limitations (e.g., multi-device imaging, modality alignment, and finite spectral bands) of regular RGB and multi-modal video tracking. However, existing hyperspectral tracking methods frequently encounter issues including data anxiety, band gap, huge volume, and weakness of the temporal condition embedded in video sequences, which result in unsatisfactory tracking capabilities. To tackle the dilemmas, we present a unified hyperspectral video tracking framework via modeling spectral-spatial-temporal conditions end-to-end, dubbed SSTtrack. First, we design the multi-modal generation adapter (MGA) to explore the interpretability benefits of combining physical and machine models for learning the multi-modal generation and bridging the band gap. To dynamically transfer and interact with multiple modalities, we then construct a novel spectral-spatial adapter (SSA). Finally, we design a temporal condition adapter (TCA) for injecting the temporal condition to guide spectral and spatial feature representations to capture static and instantaneous object properties. SSTtrack follows the prompt learning paradigm with the addition of few trainable parameters (0.575 M), resulting in superior performance in extensive comparisons. The code will be released at https://github.com/YZCU/SSTtrack.

A Smith Predictor Modified with a Pseudo Feedforward Control for the Charge-Coupled Device-Based Optoelectronic Tracking System.

In the high-precision optoelectronic tracking system (OTS) based on a charge-coupled device (CCD), the boresight error extracted from the tracking image contains an undeniable delay, which directly limits the control bandwidth of visual tracking. High bandwidth means high response speed and tracking accuracy. Generally, a model-based delay compensation control method called the Smith predictor is utilized to separate time delay from the closed loop to promote the control bandwidth. However, due to the existence of errors between the established model and the real object, the improvement in the bandwidth is still limited to ensure system stability, resulting in insufficient tracking performance. In this paper, to solve the problem, a Smith predictor modified with pseudo feedforward control for the OTS is proposed. The experimental results demonstrate that the proposed method achieves significant improvements in tracking performance, reducing the maximum residual error at 1 Hz from 365 arcseconds (using the classic Smith predictor) to 283 arcseconds, a 22.5% improvement. Across the main frequency band (0.2 Hz to 2 Hz), the residual errors were consistently lower using the proposed method.

Tropical cyclone tracking from geostationary infrared satellite images using deep learning techniques

ABSTRACT Accurate tracking of tropical cyclones (TCs) can provide regions of interest for intelligent forecasting of TC tracks and intensity. There have been few studies on algorithms for automatic TC tracking. This study proposes an effective TC tracking method based on deep learning combined with infrared satellite images. The study first constructed a TC tracking dataset based on the infrared images of the China Fengyun-2D geostationary satellite covering six different TC intensity levels between 2009 and 2012. This included 47 complete cases (video sequences) of TCs from generation to extinction. Based on deep learning, the visual tracking algorithm SiamRPN was used as the model framework. Combining Bi-GRU and TC cloud spatiotemporal evolution characteristics to improve the performance of the SiamRPN network, the SiamTCNet target-tracking model was designed to track TCs automatically. Considering that the shape and scale of TC changes with time, a TC is regarded as a typical non-rigid object with obvious timing characteristics, so the first frame of a TC video sequence is combined with the satellite images of the first three frames of the current frame as inputs to the proposed SiamTCNet model, which then extracts the evolution of the TC’s spatial structure and its bidirectional temporal change information. The experimental results show that the TC tracking of the proposed model is a significant improvement over the original SiamRPN model.

Glial expression of Drosophila UBE3A causes spontaneous seizures that can be modulated by 5-HT signaling

Misexpression of the E3 ubiquitin ligase gene UBE3A is thought to contribute to a range of neurological disorders. In the context of Dup15q syndrome, additional genomic copies of UBE3A give rise to the autism, muscle hypotonia and spontaneous seizures characteristics of the disorder. In a Drosophila model of Dup 15q syndrome, it was recently shown that glial-driven expression of the UBE3A ortholog dube3a led to a “bang-sensitive” phenotype, where mechanical shock triggers convulsions, suggesting glial dube3a expression contributes to hyperexcitability in flies. Here we directly compare the consequences of glial- and neuronal-driven dube3a expression on motor coordination and seizure susceptibility in Drosophila.To quantify seizure-related behavioral events, we developed and trained a hidden Markov model that identified these events based on automated video tracking of fly locomotion. Both glial and neuronal driven dube3a expression led to clear motor phenotypes. However, only glial-driven dube3a expression displayed spontaneous seizure-associated immobilization events, that were clearly observed at high-temperature (38 °C). Using a tethered fly preparation amenable to electrophysiological monitoring of seizure activity, we found glial-driven dube3a flies display aberrant spontaneous spike discharges which are bilaterally synchronized. Neither neuronal-dube3a overexpressing flies, nor control flies displayed these firing patterns. We previously performed a drug screen for FDA approved compounds that can suppress bang-sensitivity in glial-driven dube3a expressing flies and identified certain 5-HT modulators as strong seizure suppressors. Here we found glial-driven dube3a flies fed the serotonin reuptake inhibitor vortioxetine and the 5-HT2A antagonist ketanserin displayed reduced immobilization and spike bursting, consistent with the previous study. Together these findings highlight the potential for glial pathophysiology to drive Dup15q syndrome-related seizure activity.

Short-term responses of spider mites inform mechanisms of maize resistance to a generalist herbivore

Plants are attacked by diverse herbivorous pests with different host specializations. While host plant resistance influences pest pressure, how resistance impacts the behaviors of generalist and specialist herbivores, and the relationship to resistance, is less well known. Here, we investigated the short-term (< 1 h) behavioral changes of a generalist herbivore, the two-spotted spider mite (TSM), and a specialist herbivore, the Banks grass mite (BGM), after introduction to no-choice Tanglefoot leaf-arenas (2 × 2 cm) of three maize inbred lines (B73, B75, and B96). The widely-used inbred line B73 is susceptible to spider mites, while B75 and B96 are known to be mite resistant, especially to TSM. Video tracking was used to record TSM and BGM walking, probing, feeding, resting, web-building and travel distance on arenas of each line. Mite oviposition was also recorded after 72 h. B75, a resistant line, decreased the feeding behavior (i.e., time) of both mite species compared to B73 (susceptible control) and B96. Moreover, TSM appeared to be sensitive to both resistant lines (B75 and B96) with reduced oviposition, and increased resting and web-building times compared to susceptible B73. In contrast, the specialist BGM showed no difference in oviposition, resting and web-building time across all maize inbred lines. Our findings of quite broad and short-term responses of TSM to B75 and B96 are consistent with a role for constitutive or rapidly induced plant defenses in maize in conferring TSM resistance. Other mechanisms of plant resistance may be needed, however, for defense against specialists like BGM.

SiamS3C: spatial-channel cross-correlation for visual tracking with centerness-guided regression

SiamS3C: spatial-channel cross-correlation for visual tracking with centerness-guided regression

Visual Tracking of Hydrogen Sulfide: Application of a Novel Lysosome-Targeted Fluorescent Probe for Bioimaging and Food Safety Assessment.

The equilibrium state of hydrogen sulfide (H2S), a gaseous signaling molecule produced by lysosomal metabolites, in vivo is crucial for cellular function. Abnormal fluctuations in H2S concentration can interfere with the normal function of lysosomes, which has been closely linked to the pathogenesis of a variety of diseases. In view of this, a novel fluorescent probe Lyso-DPP based on 1,3,5-triarylpyrazolines was developed for the precise detection of H2S in lysosomes by using the hydrophilic morpholine moiety as a lysosomal targeting unit, and 2,4-dinitroanisole as a fluorescence-quenching and H2S-responsive unit. The probe cleverly combines the advantages of simple synthesis, sensitive blue fluorescence turn-on with a limit of detection, LOD, of 97.3 nM, good stability, and fast response time (10 min), which makes Lyso-DPP successful in portable monitoring of meat freshness in the form of test strips. Moreover, the excellent biocompatibility and precise targeting capability of the probe Lyso-DPP make it perform well in the monitoring of H2S in lysosomes, living cells, and zebrafish. This work not only provides new technical tools for food quality control but also paves up new ideas for early diagnosis and treatment of H2S-related diseases.

Spatial attention inference model for cascaded siamese tracking with dynamic residual update strategy

Target representation is crucial for visual tracking. Most Siamese-based trackers try their best to establish target models by using various deep networks. However, they neglect the exploration of correlation among features, which leads to the inability to learn more representative features. In this paper, we propose a spatial attention inference model for cascaded Siamese tracking with dynamic residual update strategy. First, a spatial attention inference model is constructed. The model fuses interlayer multi-scale features generated by dilation convolution to enhance the spatial representation ability of features. On this basis, we use self-attention to capture interaction between target and context, and use cross-attention to aggregate interdependencies between target and background. The model infers potential feature information by exploiting the correlations among features for building better appearance models. Second, a cascaded localization-aware network is introduced to bridge a gap between classification and regression. We propose an alignment-aware branch to resample and learn object-aware features from the predicted bounding boxes for obtaining localization confidence, which is used to correct the classification confidence by weighted integration. This cascaded strategy alleviates the misalignment problem between classification and regression. Finally, a dynamic residual update strategy is proposed. This strategy utilizes the Context Fusion Network (CFNet) to fuse the templates of historical and current frames to generate the optimal templates. Meanwhile, we use a dynamic threshold function to determine when to update by judging the tracking results. The strategy uses temporal context to fully explore the intrinsic properties of the target, which enhances the adaptability to changes in the target’s appearance. We conducted extensive experiments on seven tracking benchmarks, including OTB100, UAV123, TC128, VOT2016, VOT2018, GOT10k and LaSOT, to validate the effectiveness of our proposed algorithm.

UAV Visual Tracking with Enhanced Feature Information

Unmanned aerial vehicles (UAVs) visual tracking is an important research direction. The tracking object is lost due to the problems of target occlusion, illumination variation, flight vibration and so on. Therefore, based on a Siamese network, this study proposes a UAVs visual tracker named SiamDFT++ to enhance the correlation of depth features. First, the network width of the three-layer convolution after the full convolution neural network is doubled, and the appearance information of the target is fully utilized to complete the feature extraction of the template frame and the detection frame. Then, the attention information fusion module and feature deep convolution module are proposed in the template branch and the detection branch, respectively. The feature correlation calculation methods of the two depths can effectively suppress the background information, enhance the correlation between pixel pairs, and efficiently complete the tasks of classification and regression. Furthermore, this study makes full use of shallow features to enhance the extraction of object features. Finally, this study uses the methods of deep cross-correlation operation and complete intersection over union to complete the matching and location tasks. The experimental results show that the tracker has strong robustness in UAVs short-term tracking scenes and long-term tracking scenes.

MIMTracking: Masked image modeling enhanced vision transformer for visual object tracking

Recently, one-stage trackers achieve state-of-the-art tracking results due to more sufficient integration of search and template representations. These methods usually adopt an encoder for synchronous feature generation and interaction. Despite their high performance, the one-stage approaches tend to feed the encoder full input representations that are highly redundant and dense during training. With a focus on tackling this problem, a novel algorithm termed MIMTracking is developed for visual target tracking. MIMTracking exploits an encoder and a decoder for masked image modeling during training. Randomly sampled discrete search embeddings and template embeddings serve as input of the encoder. The lightweight decoder takes full representations as input and progressively highlights the target region. This design alleviates input redundancy and reduces the computational cost of the training process, thereby allowing for more efficient learning of useful representations. The proposed MIMTracking achieves state-of-the-art tracking results on numerous tracking datasets, e.g., 51.5 % area under curve (AUC) on LaSOText, outperforming the previous top tracker OSTrack by 2 %. Especially, our large model MIMTracking-L further improves the AUC to 53.4 % on LaSOText.

FQTrack:Object Tracking Method Based on a Feature-Enhanced Memory Network and Memory Quality Selection Mechanism

Visual object tracking technology is widely used in intelligent security, automatic driving and other fields, and also plays an important role in frontier fields such as human–computer interactions and virtual reality. The memory network improves the stability and accuracy of tracking by using historical frame information to assist in the positioning of the current frame in object tracking. However, the memory network is still insufficient in feature mining and the accuracy and robustness of the model may be reduced when using noisy observation samples to update it. In view of the above problems, we propose a new tracking framework, which uses the attention mechanism to establish a feature-enhanced memory network and combines cross-attention to aggregate the spatial and temporal context information of the target. The former introduces spatio-temporal adaptive attention and cross-spatial attention, embeds spatial location information into channels, realizes multi-scale feature fusion, dynamically emphasizes target location information, and obtains richer feature maps. The latter guides the tracker to focus on the area with the largest amount of information in the current frame to better distinguish the foreground and background. In addition, through the memory quality selection mechanism, the accuracy and richness of the feature samples are improved, thereby enhancing the adaptability and discrimination ability of the tracking model. Experiments on benchmark test sets such as OTB2015, TrackingNet, GOT-10k, LaSOT and UAV 123 show that this method achieves comparable performance with advanced trackers.

Eye tracking study in children to assess mental calculation and eye movements

Eye tracking technology is a high-potential tool for different mathematic cognition research areas. Moreover, there is a dire need for more studies that provide detailed information on the quality of registered eye data. This study aimed to illustrate the applicability of eye tracking in the examination of mathematical cognition, focusing specifically on primary school students completing a computerized mental arithmetic task. Results suggested that the eye tracking device effectively captured high-quality eye movement data when primary school children engaged in this specific task. Furthermore, significant negative correlations have been found between task performance and number of eye fixations. Finally, eye movements distinctions between “Areas of Interest” have been found, indicating different visual tracking associated with different components of arithmetic calculations. This study underscores the extensive possibilities for future research employing eye tracking devices during computerized calculation tasks as assessment tools to explore the complex visual and cognitive processes.

Walking representation and simulation based on multi-source image fusion and multi-agent reinforcement learning for gait rehabilitation

In the formulation of strategies for walking rehabilitation, achieving precise identification of the current state and making rational predictions about the future state are crucial but often unrealized. To tackle this challenge, our study introduces a unified framework that integrates a novel 3D walking motion capture method using multi-source image fusion and a walking rehabilitation simulation approach based on multi-agent reinforcement learning. We found that, (i) the proposal achieved an accurate 3D walking motion capture and outperforms other advanced methods. Experimental evidence indicates that, compared to similar visual skeleton tracking methods, the proposed approach yields results with higher Pearson correlation (r=0.93), intra-class correlation coefficient (ICC(2,1)=0.91), and narrower confidence intervals ([0.90,0.95] for r, [0.88,0.94] for ICC(2,1)) when compared to standard results. The outcomes of the proposed approach also exhibit commendable correlation and concurrence with those obtained through the IMU-based skeleton tracking method in the assessment of gait parameters ([0.85,0.89] for r, [0.75,0.81] for ICC(2,1)); (ii) multi-agent reinforcement learning has the potential to be used to solve the simulation task of gait rehabilitation. In mimicry experiment, our proposed simulation method for gait rehabilitation not only enables the intelligent agent to converge from the initial state to the target state, but also observes evolutionary patterns similar to those observed in clinical practice through motor state resolution. This study offers valuable contributions to walking rehabilitation, enabling precise assessment and simulation-based interventions, with potential implications for clinical practice and patient outcomes.

Experimental study of the wall particle motion in a screw reactor

Screw reactors are gas/solid systems widely employed in industrial applications. Understanding the local powder mixing within these reactors is crucial, particularly at the reactor wall where wall/solid heat transfer occurs. This study proposes a non-invasive measurement method utilizing optical devices to assess wall particle motion. Two techniques for evaluating the particle motion are compared: a simple visual tracking method and particle image velocimetry. Both methods yield similar results, demonstrating consistent wall particle motion across the tube periphery during experimentation. The influence of operational and geometric parameters has been investigated, revealing that filling degree, powder flowability, and screw geometry significantly impact wall particle motion. Symbolic regression has been employed to derive dimensionless models that effectively predict experimental data. These models can be utilized to forecast wall particle motion, serving as a valuable tool for screw conveyor design.