Search Image Research Articles

A transformer based visual tracker with restricted token interaction and knowledge distillation

Recently, one-stream pipelines have made significant progress in visual object tracking (VOT), where the template and search images interact in early stages. However, one-stream pipelines have a potential problem: They treat the object and the background equally (or other irrelevant parts), leading to weak discriminability of the extracted features. To remedy this issue, a restricted token interaction module based on asymmetric attention mechanism is proposed in this paper, which divides the search image into valuable part and other part. Only the valuable part is selected for cross-attention with the template so as to better distinguish the object from the background, which finally improves the localization accuracy and robustness. In addition, to avoid heavy computational overhead, we utilize logit distillation and localization distillation methods to optimize the outputs of the classification and regression heads respectively. At the same time, we separate the distillation regions and apply different knowledge distillation methods in different regions to effectively determine which regions are most beneficial for classification or localization learning. Extensive experiments have been conducted on mainstream datasets in which our tracker (dubbed RIDTrack) has achieved appealing results while meeting the real-time requirement.

Detecting a Stalker: The Effect of Body Posture, Gaze Direction, and Camouflage Pattern on Predator Detection.

The success of a predatory attack is related to how much a predator manages to approach a prey without being detected. Some carnivore mammals use environmental objects (e.g., leaves and branches) as visual obstacles during stalking behavior, allowing them to expose only parts of their bodies while approaching and visual monitoring their prey. Here, we investigate the influence of carnivores' body postures, gaze direction, and camouflage pattern on their detection by prey. To do so, we photographed taxidermized carnivore models (cougar, ocelot, and lesser grison) in their preserved natural habitats and presented these images to human dichromats (i.e., colourblinds) and trichromats (i.e., normal color vision). Our findings highlight the importance of body outline and gaze as search images during predator detection tasks. We also demonstrate that coat and facial color patterns can camouflage predator's body outline and gaze. This is the first behavioral evidence that the facial coloration of natural predators might mask their gaze to potential prey. Furthermore, we observed that carnivore coat color patterns may serve as an additional cue for trichromats, particularly in hidden carnivore detection tasks that proved to be more challenging for dichromats. Our results show possible strategies that evolved between predator and preys, in which prey make use of body outlines, gaze direction, and coat color to improve predator detection, while predators potentially evolved stalking behavior and body/gaze camouflage as counter strategies to cope with the improvements in prey's perception and conceal their presence.

Prey detection by a stepwise visual template matching mechanism.

Predators can improve prey capture using a search image, and recent prey provide a visual template with which subsequent prey are compared. Considering trout feeding responses to mayfly prey of different sizes and phenological availability across years, we tested if changing relative abundances (ratios) of prey of the same species, but different body sizes, shifted trout feeding behaviour. For example, we hypothesized that a feeding switch from larger to smaller prey required continuous exposure to the novel smaller prey. The hypothesis that continuous exposure to novel small prey results in their acceptance was not supported. Rather, we discovered that trout identify novel prey using a dynamic stepwise visual neural template prey matching process, which involves the formation of focal prey template based on size or type, rejection of novel prey that do not match the size or type templates and modification of the existing or development of multiple prey templates that eventually enabled recognition of novel, small prey. We also discovered trout store multiple visual prey templates in memory. These results have implications for predator and prey dynamics, optimal foraging, the persistence of rare prey, prey species coexistence and predator selection on prey phenology.

Design of intelligent algorithm for object search based on IoT digital images

With the development of artificial intelligence, traditional object search and image recognition have been replaced by the Internet of Things and artificial intelligence. However, traditional object search algorithms often lack accuracy and low precision. Therefore, this study proposes a new intelligent encryption algorithm to address the issues of insufficient accuracy in object search algorithms and image recognition algorithms. The new algorithm ensures the security of user data and the response efficiency of the model during the conversation process by integrating fully homomorphic encryption technology and dynamic sparse attention mechanism. The dynamic sparse attention mechanism introduced simultaneously improves the model's ability to handle long sequence data by dynamically adjusting attention weights. Experimental results showed that the precision of the proposed algorithm was 0.05 % higher than that of random algorithms and 0.19 % higher than that of sorting algorithms. The recall rate of the proposed algorithm was 0.14 % higher than that of random algorithms and 0.16 % higher than that of sorting algorithms. The research algorithm can identify objects with certain characteristics and is suitable for specific environments, greatly reducing the probability of data leakage in object search and providing new ideas for research in this field.

The Dark Energy Survey Supernova Program: Light Curves and 5 Yr Data Release

We present griz photometric light curves for the full 5 yr of the Dark Energy Survey Supernova (DES-SN) program, obtained with both forced point-spread function photometry on difference images (DiffImg) performed during survey operations, and scene modelling photometry (SMP) on search images processed after the survey. This release contains 31,636 DiffImg and 19,706 high-quality SMP light curves, the latter of which contain 1635 photometrically classified SNe that pass cosmology quality cuts. This sample spans the largest redshift (z) range ever covered by a single SN survey (0.1 < z < 1.13) and is the largest single sample from a single instrument of SNe ever used for cosmological constraints. We describe in detail the improvements made to obtain the final DES-SN photometry and provide a comparison to what was used in the 3 yr DES-SN spectroscopically confirmed Type Ia SN sample. We also include a comparative analysis of the performance of the SMP photometry with respect to the real-time DiffImg forced photometry and find that SMP photometry is more precise, more accurate, and less sensitive to the host-galaxy surface brightness anomaly. The public release of the light curves and ancillary data can be found at github.com/des-science/DES-SN5YR and doi:10.5281/zenodo.12720777.

A novel deformation measurement method for rotating blade based on PSO-ILS image correlation matching and mismatch correction

Abstract Aiming at the problem of small measurement range and difficult measurement of traditional contact sensor method in the rotating state of aero-engine blades, this paper proposes a novel deformation measurement method for rotating blade based on image correlation matching and mismatch correction. Firstly, a Particle Swarm Optimization-Iterative Local Search image intelligent matching algorithm is proposed, which effectively balances global search and local optimization, and the image matching displacement accuracy reaches 10−3 pixel. Secondly, a mismatch point detection method based on multi-scale local Root Mean Square is proposed, and the high-precision detection of mismatch points is realized by considering the influence of local structure. Finally, through the hierarchical refinement of the sub-pixel level mismatch point correction method, the mismatch point is corrected in the two iterative stages of global search and local optimization. The corrected mismatch point response value K is reduced by 99% compared with that before correction, which further improves the accuracy of deformation field calculation. In the experiment, the deformation of the rotating blade of 4500 RPM is measured, which proves that the image intelligent matching algorithm and the deformation field calculation method proposed in this paper can provide new methods and technical support for the accurate measurement of the blade surface deformation.

Experiments Reveal That Search Image Might Be Responsible for Seasonal Variation in Nest-Predation Rates

Experiments Reveal That Search Image Might Be Responsible for Seasonal Variation in Nest-Predation Rates

Совершенствование медико-социальных мер по охране психического здоровья подростков в системе первичной медико-санитарной помощи (обзор литературы).

Actuality. To develop effective tools for protecting the mental health of adolescents, medical workers need to clarify the leading reasons for its formation in modern conditions, predictors and criteria for the risk of its deterioration. Currently, there are no integrated medicalpsychological- social monitoring studies of adolescent mental health, including dynamic monitoring of psychological, pedagogical, social and medical indicative indicators. The purpose of the study is to review scientific publications devoted to the analysis of the state of mental health of children and adolescents in the Russian Federation. Research methods. Using the search image: («mental health of children and adolescents», «organization of psychiatric care for children and adolescents», «addictive behavior of children and adolescents», «risk factors for mental health of children and adolescents»), a collection of 63 relevant articles published by domestic and foreign authors over the past 15 years (2013–2023) have been formed, indexed in the bibliometric databases WoS, Scopus, RSCI, and in the search engines PubMed, Google Scholar. Research results. The performed analysis allows us to note that such issues as the manifestation of clinical mental disorders, their predictors with age-related detail, and the development of integral indicators of mental health deterioration have not been sufficiently studied. Studies indicating the presence of problems in the system of mental health protection of minors during their studies in general education organizations do not provide specific recommendations for their solution.. Conclusion. To develop effective tools for the protection of adolescent mental health, medical professionals need to clarify the leading causes of its formation in modern conditions, predictors and criteria for the risk of its deterioration. This will make it possible to fill the existing deficit in the algorithms of medical support for adolescents during their studies in general education organizations and medical, psychological and pedagogical integration.

Highly compact adaptive network based on transformer for RGBT tracking

RGBT tracking is a challenging task that requires robust fusion of visible images (RGB) and thermal infrared images (TIR) to handle various scenarios, such as illumination changes, occlusions, and camouflage. The current popular RGBT trackers mostly based on two stream Siamese trackers. They tend to separately extracts template and search images region features, which neglects relationship between target and background. Moreover, they do not fuse RGB and TIR features properly, limiting their ability to utilize complementary features. To address these issues, we introduce a highly compact adaptive transformer-based network that unifies the process of feature extracting and correlation between template and search images. In this way, the compact network has dual-branches. They can combine feature extraction and correlation for different modalities of template images and search images. Meanwhile, we introduce a cross-modal weight redistribution module (CMWR) for multi-modal fusion. This adaptive fusion scheme learns discriminative features of RGB and TIR data and assigns weights to them, enabling them to complement each other. Furthermore, to address the issue of tracking targets of different scales, we design scale-adaptive optimization pyramid module (SAOP) that adapt to objects of different sizes. Our method achieves exceptional performance on the GTOT, RGBT234 and LasHeR datasets, surpassing most of the existing methods. The results are consistent across multiple datasets, demonstrating the effectiveness and superiority of our approach. And our code is released at: https://github.com/ELOESZHANG/HCANet.

Color polymorphism and mating trends in a population of the alpine leaf beetle Oreina gloriosa.

The bright colors of Alpine leaf beetles (Coleoptera, Chrysomelidae) are thought to act as aposematic signals against predation. Within the European Alps, at least six species display a basal color of either blue or green, likely configuring a classic case of müllerian mimicry. In this context, intra-population color polymorphism is paradoxical as the existence of numerous color morphs might hamper the establishment of a search image in visual predators. Assortative mating may be one of the main factors contributing to the maintenance of polymorphic populations. Due to the marked iridescence of these leaf beetles, the perceived color may change as the viewing or illumination angle changes. The present study, conducted over three years, involved intensive sampling of a population of Oreina gloriosa from the Italian Alps and applied colorimetry and a decision tree method to identify the color morphs in an objective manner. The tertiary sex ratio of the population was biased in favor of males, suggesting that viviparous females hide to give birth. Seven color morphs were identified, and their frequencies varied significantly over the course of the study. Three different analyses of mating (JMating, QInfomating, and Montecarlo simulations) recognized a general trend for random mating which coexists with some instances of positive and negative assortative mating. This could help explain the pre-eminence of one morph (which would be favored because of positive selection due to positive assortative mating) in parallel with the persistence of six other morphs (maintained due to negative assortative mating).

SEER: Backdoor Detection for Vision-Language Models through Searching Target Text and Image Trigger Jointly

This paper proposes SEER, a novel backdoor detection algorithm for vision-language models, addressing the gap in the literature on multi-modal backdoor detection. While backdoor detection in single-modal models has been well studied, the investigation of such defenses in multi-modal models remains limited. Existing backdoor defense mechanisms cannot be directly applied to multi-modal settings due to their increased complexity and search space explosion. In this paper, we propose to detect backdoors in vision-language models by jointly searching image triggers and malicious target texts in feature space shared by vision and language modalities. Our extensive experiments demonstrate that SEER can achieve over 92% detection rate on backdoor detection in vision-language models in various settings without accessing training data or knowledge of downstream tasks.

Sequential Fusion Based Multi-Granularity Consistency for Space-Time Transformer Tracking

Regarded as a template-matching task for a long time, visual object tracking has witnessed significant progress in space-wise exploration. However, since tracking is performed on videos with substantial time-wise information, it is important to simultaneously mine the temporal contexts which have not yet been deeply explored. Previous supervised works mostly consider template reform as the breakthrough point, but they are often limited by additional computational burdens or the quality of chosen templates. To address this issue, we propose a Space-Time Consistent Transformer Tracker (STCFormer), which uses a sequential fusion framework with multi-granularity consistency constraints to learn spatiotemporal context information. We design a sequential fusion framework that recombines template and search images based on tracking results from chronological frames, fusing updated tracking states in training. To further overcome the over-reliance on the fixed template without increasing computational complexity, we design three space-time consistent constraints: Label Consistency Loss (LCL) for label-level consistency, Attention Consistency Loss (ACL) for patch-level ROI consistency, and Semantic Consistency Loss (SCL) for feature-level semantic consistency. Specifically, in ACL and SCL, the label information is used to constrain the attention and feature consistency of the target and the background, respectively, to avoid mutual interference. Extensive experiments have shown that our STCFormer outperforms many of the best-performing trackers on several popular benchmarks.

DPT‐tracker: Dual pooling transformer for efficient visual tracking

AbstractTransformer tracking always takes paired template and search images as encoder input and conduct feature extraction and target‐search feature correlation by self and/or cross attention operations, thus the model complexity will grow quadratically with the number of input images. To alleviate the burden of this tracking paradigm and facilitate practical deployment of Transformer‐based trackers, we propose a dual pooling transformer tracking framework, dubbed as DPT, which consists of three components: a simple yet efficient spatiotemporal attention model (SAM), a mutual correlation pooling Transformer (MCPT) and a multiscale aggregation pooling Transformer (MAPT). SAM is designed to gracefully aggregates temporal dynamics and spatial appearance information of multi‐frame templates along space‐time dimensions. MCPT aims to capture multi‐scale pooled and correlated contextual features, which is followed by MAPT that aggregates multi‐scale features into a unified feature representation for tracking prediction. DPT tracker achieves AUC score of 69.5 on LaSOT and precision score of 82.8 on TrackingNet while maintaining a shorter sequence length of attention tokens, fewer parameters and FLOPs compared to existing state‐of‐the‐art (SOTA) Transformer tracking methods. Extensive experiments demonstrate that DPT tracker yields a strong real‐time tracking baseline with a good trade‐off between tracking performance and inference efficiency.

SiamRAAN: Siamese Residual Attentional Aggregation Network for Visual Object Tracking

The Siamese network-based tracker calculates object templates and search images independently, and the template features are not updated online when performing object tracking. Adapting to interference scenarios with performance-guaranteed tracking accuracy when background clutter, illumination variation or partial occlusion occurs in the search area is a challenging task. To effectively address the issue with the abovementioned interference and to improve location accuracy, this paper devises a Siamese residual attentional aggregation network framework for self-adaptive feature implicit updating. First, SiamRAAN introduces Self-RAAN into the backbone network by applying residual self-attention to extract effective objective features. Then, we introduce Cross-RAAN to update the template features online by focusing on the high-relevance parts in the feature extraction process of both the object template and search image. Finally, a multilevel feature fusion module is introduced to fuse the RAAN-enhanced feature information and improve the network’s ability to perceive key features. Extensive experiments conducted on benchmark datasets (GOT-10K, LaSOT, OTB-50, OTB-100 and UAV123) demonstrated that our SiamRAAN delivers excellent performance and runs at 51 FPS in various challenging object tracking tasks. Code is available at https://github.com/MallowYi/SiamRAAN.

Habitat heterogeneity limits prey colour polymorphism maintained via negative frequency-dependent selection.

The persistence of non-neutral trait polymorphism is enigmatic because stabilizing selection is expected to deplete variation. In cryptically coloured prey, negative frequency-dependent selection due to search image formation by predators has been proposed to favour rare variants, promoting polymorphism. However, in a heterogeneous environment, locally varying disruptive selection favours patch type-specific optima, resulting in spatial segregation of colour variants. Here, we address whether negative frequency-dependent selection can overcome selection posed by habitat heterogeneity to promote local polymorphism using an individual-based model. In addition, we compare how prey and predator mobility may modify the outcome. Our model revealed that frequency-dependent predation could strongly promote local prey polymorphism, but only when differences between morphs in patch-specific fitness were small. The effect of frequency-dependent predation depended on the predator adjustment of search image and was hampered by the prey population structure. Gene flow due to prey movement counteracted local selection, promoted local polymorphism to some extent, and relaxed the conditions for polymorphism due to frequency-dependent predation. Importantly, abrupt spatial changes in morph frequencies decreased the probability that mobile frequency-dependent predators could maintain local prey polymorphism. Overall, our study suggests that in a spatially heterogeneous environment, negative frequency-dependent selection may help maintain local polymorphism but only under a limited range of conditions.

Human Detection From Unmanned Aerial Vehicles’ Images for Search and Rescue Missions: A State-of-the-Art Review

Human Detection From Unmanned Aerial Vehicles’ Images for Search and Rescue Missions: A State-of-the-Art Review

Statistical similarity matching and filtering for clinical image retrieval by machine learning approach

Clinical image retrieval plays a pivotal role in modern healthcare for diagnostics and research, but prior research has grappled with the challenge of achieving high accuracy due to limited filtering techniques. The proposed method includes statistical distance measurements for similarity comparison and a machine learning technique for image filtering. Throughout this framework, the search area for similarity matching is reduced by first filtering away irrelevant images using the probabilistic outcomes of the Support Vector Machine (SVM) classification as class predictions of search and database images. Resizing is done as part of the preprocessing. Then, using Principal Component Analysis (PCA), the preprocessed data’s textural features, visual characteristics, and low-level features are extracted. The study also suggested an adaptive similarity matching method centered on a linear integration of feature-level similarities on the individual-level level. The precision and ranking order details of the most appropriate images retrieved and predicted by SVMs are considered when calculating the feature weights. The system continually alters weights for every distinctive search to generate beneficial outcomes. The supervised and unsupervised learning strategies are studied to link low-level global image features in the generated PCA-based Eigen Space using their high-level semantic and visual classifications to reduce the semantic gap and enhance retrieval effectiveness. The ground-truth database used in experiments has 1594 unique medical images with 3 different databases. Our method significantly improves the precision and recall rates in image retrieval tasks by combining sophisticated feature extraction, data-driven algorithms, and deep learning models. Research obtained an impressive accuracy of 0.99, demonstrating the effectiveness of our approach. This novel methodology addresses the limitations of prior research and provides a robust and reliable solution for clinicians and researchers in the medical field seeking to access and analyze relevant clinical images.

Mutually exclusive feeding yields Holling type III functional response

Abstract Many generalist predators, including fish, insects and mammals, temporarily focus on their search behaviour and can switch between prey species. Flexible and rapid behavioural plasticity enables a predator to forage optimally, as guided by search image formation and prey switching in response to changing prey abundances. More than half a century of studies have used Holling type III functional response to model the impact of this feeding behaviour on population dynamics, yet a mechanistic understanding for the pathway between individual behaviour and feeding rates remains poorly understood. To understand this common feeding behaviour, we developed three mechanistic models using first‐principles to provide direct derivations for the relationship between observed individual behaviour and feeding rates. The derived models yield three novel, generalised functional responses for predators characterised by prey switching and mutually exclusive feeding (i.e. feeding is limited to one prey species at a time). We show how these functional responses act as Holling type III response, and how they can be used to predict predators' diet compositions. Our results demonstrate that several behavioural forms of prey switching, as displayed in a wide range of predator species, consistently produce type III conforming functional responses. Read the free Plain Language Summary for this article on the Journal blog.

Exploring reliable infrared object tracking with spatio-temporal fusion transformer

Currently, most thermal infrared (TIR) trackers rely on feature matching between the search image and a fixed template cropped from the first frame. Some Siam-based TIR trackers with a template update mechanism introduce historical prediction information in the temporal dimension through correlation filters. However, their feature characterization capability is inadequate to resist target scale variations, appearance changes, and occlusion. To address this challenge, we explore a novel spatio-temporal fusion Transformer (STFT) model to realize robust TIR object tracking. Our approach involves a Transformer-based encoder–decoder that fuses spatio-temporal information. Specifically, we design a dynamic template update strategy based on salient points feature(SPF) representation, which allows the model to leverage the most powerful spatio-temporal information by retrieving multiple salient points on the target image. To further fortify the dynamic template update strategy, we propose an IoU-Aware target state estimation head that utilizes the joint representation of target classification and localization. An IoU-Aware criterion is developed for quality estimation of the dynamic template. The proposed STFT-Net approach has been put to the evaluation on three challenging benchmarks, with extensive experimental results showcasing its superior performance in contrast to acclaimed tracking algorithms. The code is available at https://github.com/qinxin-wh/STFT-Net.

RETRACTED ARTICLE: Dual Siamese Anchor Points Adaptive Tracker with Transformer for RGBT Tracking

Due to environmental conditions, such as rainy days, foggy days, and dim lighting, objects in visible light images are not prominently displayed, leading to an easy loss of targets during tracking. In recent years, many RGB visible light trackers have achieved significant success in addressing visual tracking challenges. However, these trackers perform poorly when tracking targets under special conditions, such as occlusions and low-light scenarios. In contrast, objects in thermal infrared images are more distinct in poor lighting conditions. Given this characteristic, researchers have shown increased interest in the development of trackers that combine thermal infrared and visible light imagery. However, some mainstream RGBT (red–green–blue and thermal) algorithms, such as MANET and ADNET, are based on the anchor-based theory, requiring consideration of anchor box sizes and introducing a substantial number of hyperparameters. This can lead to suboptimal performance when tracking dynamically changing targets. Moreover, these models rely on convolutional neural networks for feature extraction, which have limitations in capturing global features. In this paper, we introduce a novel training network model called DAPAT, which combines the anchor-free concept with Transformer theory. DAPAT differs from previous models in several ways. Specifically, we have designed a straightforward model to extract precise global features from template and search images. We have also incorporated two enhancement modules into the model to improve template and search images of different sizes while suppressing the impact of non-target images. We employ a dual-stream feature fusion network to reduce the loss of image feature information due to feature correlation operations. Finally, we compare the performance of the tracking model proposed in this paper with some advanced RGBT trackers on three data sets (RGBT234, RGBT210, and GTOT). The test results demonstrate that our tracker exhibits improvements in robustness and success rate, among other performance aspects.