Siamese Network Research Articles

Contextual Enhancement–Interaction and Multi-Scale Weighted Fusion Network for Aerial Tracking

Siamese-based trackers have been widely utilized in UAV visual tracking due to their outstanding performance. However, UAV visual tracking encounters numerous challenges, such as similar targets, scale variations, and background clutter. Existing Siamese trackers face two significant issues: firstly, they rely on single-branch features, limiting their ability to achieve long-term and accurate aerial tracking. Secondly, current tracking algorithms treat multi-level similarity responses equally, making it difficult to ensure tracking accuracy in complex airborne environments. To tackle these challenges, we propose a novel UAV tracking Siamese network named the contextual enhancement–interaction and multi-scale weighted fusion network, which is designed to improve aerial tracking performance. Firstly, we designed a contextual enhancement–interaction module to improve feature representation. This module effectively facilitates the interaction between the template and search branches and strengthens the features of each branch in parallel. Specifically, a cross-attention mechanism within the module integrates the branch information effectively. The parallel Transformer-based enhancement structure improves the feature saliency significantly. Additionally, we designed an efficient multi-scale weighted fusion module that adaptively weights the correlation response maps across different feature scales. This module fully utilizes the global similarity response between the template and the search area, enhancing feature distinctiveness and improving tracking results. We conducted experiments using several state-of-the-art trackers on aerial tracking benchmarks, including DTB70, UAV123, UAV20L, and UAV123@10fps, to validate the efficacy of the proposed network. The experimental results demonstrate that our tracker performs effectively in complex aerial tracking scenarios and competes well with state-of-the-art trackers.

Semantic segmentation algorithm for video from UAV based on adaptive keyframe scheduling via similarity measurement

Abstract Unmanned aerial vehicle (UAV) videos exhibit complex object motion features and significant differences between frame features. To solve the problems of feature information loss and drastic accuracy decline in applying the video semantic segmentation method based on the fixed-period update strategy of keyframes, a keyframe recognition method based on similarity measurement is proposed, forming a video semantic segmentation algorithm based on adaptive keyframe scheduling. one keyframe recognition method based on pixel similarity measurement is established by modeling the similarity relationship between low-level pixels in adjacent frames. Meanwhile, the other keyframe recognition method based on feature similarity measurement is established by constructing a shallow Siamese network to measure the similarity relationship between features of frames. Then a discriminative network is constructed based on the obtained similarity of inter frames, and the segmentation process is accelerated by reusing features of keyframes through the optical flow network. Thereby a video semantic segmentation method for UAV based on adaptive keyframe strategy is established. The effectiveness of the proposed video semantic segmentation algorithm is verified on UAVid dataset. The results demonstrated that the speed of the proposed algorithm reaches 53.2 FPS and 54.5 FPS on the premise that the mean intersection over union is higher than 40% (this value is compared with the segmentation accuracy in the balanced mode when the similarity threshold is 0.76 and 0.88, respectively), which is 18.5 FPS and 19.8 FPS higher than the PSPNet image semantic segmentation algorithm. In addition, analysis of experiment results shows that the pixel-similarity-based keyframe recognition is suitable for high-precision video semantic segmentation scenes that need to improve segmentation efficiency, and the feature-similarity-based keyframe recognition is more suitable for high real-time video semantic segmentation tasks that require a small decrease in overall algorithm accuracy. 
In a word, the proposed video semantic segmentation algorithm based on adaptive keyframe scheduling via similarity measurement can improve segmentation speed while ensuring segmentation accuracy and stability.

Fine-Grained Recognition of Fidgety-Type Emotions Using Multi-Scale One-Dimensional Residual Siamese Network

Fidgety speech emotion has important research value, and many deep learning models have played a good role in feature modeling in recent years. In this paper, the problem of practical speech emotion is studied, and the improvement is made on fidgety-type emotion using a novel neural network model. First, we construct a large number of phonological features for modeling emotions. Second, the differences in fidgety speech between various groups of people were studied. Through the distribution of features, the individual features of fidgety emotion were studied. Third, we propose a fine-grained emotion classification method, which analyzes the subtle differences between emotional categories through Siamese neural networks. We propose to use multi-scale residual blocks within the network architecture, and alleviate the vanishing gradient problem. This allows the network to learn more meaningful representations of fidgety speech signal. Finally, the experimental results show that the proposed method can provide the versatility of modeling, and that fidgety emotion is well identified. It has great research value in practical applications.

Realization of Reliable and Effective Authentication in Intelligent Systems by Using Visual Biometrics Methods

The main purpose of this article is to consider the aspects of ensuring security and increasing the efficiency of the authentication process in intelligent systems using visual biometrics. The work is aimed at the development and improvement of authentication systems using advanced biometric identification methods. An intelligent system has been created that ensures secure authentication of users of the current system, using a Siamese neural network. In addition to the implementation of basic security measures in the form of hashing and saving user logins and passwords, the implementation of two-factor authentication is important nowadays, which significantly strengthens the protection of user data and prevents most modern methods of hacking and stealing user data. Two-factor authentication is implemented as a technology for searching, recognizing and comparing the faces of system users, as visual biometrics is more secure than other types of two-factor authentication. Different variations of the possible implementation of Siamese neural network using Contrastive loss function and more modern Triplet loss function were reviewed and accordingly, a neural network using Triplet loss function was accomplished and trained. After training and verifying the correct operation of the neural network, it was integrated into the created intelligent system, thanks to which an effective way of recognizing the face of the system user was created, saving the received information in the database and further comparing the current user with the stored face during authentication. As a result, a secure and reliable intelligent system was created that cutting down the risk of unapproved access to the user account and uses an effective and modern method of user authentication.

A lightweight target tracking algorithm based on online correction for meta-learning

The traditional Siamese network based object tracking algorithms suffer from high computational complexity, making them difficult to run on embedded devices. Moreover, when faced with long-term tracking tasks, their success rates significantly decline. To address these issues, we propose a lightweight long-term object tracking algorithm called Meta-Master-based Ghost Fast Tracking (MGTtracker),which based on meta-learning. This algorithm integrates the Ghost mechanism to create a lightweight backbone network called G-ResNet, which accurately extracts target features while operating quickly. We design a tiny adaptive weighted fusion feature pyramid network (TiFPN) to enhance feature information fusion and mitigate interference from similar objects. We introduce a lightweight region regression network, the Ghost Decouple Net (GDNet) for target position prediction. Finally, we propose a meta-learning-based online template correction mechanism called Meta-Master to overcome error accumulation in long-term tracking tasks and the difficulty of reacquiring targets after loss. We evaluate the algorithm on public datasets OTB100, VOT2020, VOT2018LT, and LaSOT and deploy it for performance testing on Jetson Xavier NX. Experimental results demonstrate the effectiveness and superiority of the algorithm. Compared to existing classic object tracking algorithms, our approach achieves a faster running speed of 25 FPS on NX, and real-time correction enhances the algorithm’s robustness. Although similar in accuracy and EAO metrics, our algorithm outperforms similar algorithms in speed and effectively addresses the issues of significant cumulative errors and easy target loss during tracking. Code is released at https://github.com/ygh96521/MGTtracker.git.

SiamDCFF: Dynamic Cascade Feature Fusion for Vision Tracking.

Establishing an accurate and robust feature fusion mechanism is key to enhancing the tracking performance of single-object trackers based on a Siamese network. However, the output features of the depth-wise cross-correlation feature fusion module in fully convolutional trackers based on Siamese networks cannot establish global dependencies on the feature maps of a search area. This paper proposes a dynamic cascade feature fusion (DCFF) module by introducing a local feature guidance (LFG) module and dynamic attention modules (DAMs) after the depth-wise cross-correlation module to enhance the global dependency modeling capability during the feature fusion process. In this paper, a set of verification experiments is designed to investigate whether establishing global dependencies for the features output by the depth-wise cross-correlation operation can significantly improve the performance of fully convolutional trackers based on a Siamese network, providing experimental support for rational design of the structure of a dynamic cascade feature fusion module. Secondly, we integrate the dynamic cascade feature fusion module into the tracking framework based on a Siamese network, propose SiamDCFF, and evaluate it using public datasets. Compared with the baseline model, SiamDCFF demonstrated significant improvements.

SiameseFWI: A Deep Learning Network for Enhanced Full Waveform Inversion

AbstractThe performance of full‐wave inversion (FWI) depends highly on how we compare the simulated data to observed ones. The simplified assumptions used to generate the simulated data make such comparison even harder. To address this challenge, we introduce SiameseFWI, a novel approach to FWI that plays a critical role in the comparative analysis of simulated and observed seismic data. Employing a Siamese network, this methodology transforms the data into a shared latent space, enabling a robust and effective comparison of data representations. SiameseFWI leverages two identical Convolutional Neural Networks with shared weights trained in a self‐supervised framework, eliminating the necessity for labeled data. In each FWI iteration, the Siamese network and the velocity model are updated to minimize Euclidean distance loss between the latent representations of the data. Empirical evaluation conducted on the Marmousi2 and Overthrust models affirms the robust inversion performance of SiameseFWI compared to traditional FWI methodologies. Furthermore, its application to field data from Western Australia demonstrates its strength and efficacy in inversion. Notably, SiameseFWI exhibits robust inversion performance even in the presence of noise or when employing a linear initial model.

Efficient Multi-Object Tracking Using RNN and Siamese Re-Identification

This work presents a comprehensive study and methodology to cope the problem of multi-object tracking. The methodology proposed here aims to simplify the problem space by using RNN as a regression network for predicting current object tracker information from past data and using Siamese network for re-identification. The number of re-identification is reduced by doing suboptimal perfect matching using a tree data structure for geometric location data of objects in view port.

An experimental evaluation of Siamese Neural Networks for robot localization using omnidirectional imaging in indoor environments

The objective of this paper is to address the localization problem using omnidirectional images captured by a catadioptric vision system mounted on the robot. For this purpose, we explore the potential of Siamese Neural Networks for modeling indoor environments using panoramic images as the unique source of information. Siamese Neural Networks are characterized by their ability to generate a similarity function between two input data, in this case, between two panoramic images. In this study, Siamese Neural Networks composed of two Convolutional Neural Networks (CNNs) are used. The output of each CNN is a descriptor which is used to characterize each image. The dissimilarity of the images is computed by measuring the distance between these descriptors. This fact makes Siamese Neural Networks particularly suitable to perform image retrieval tasks. First, we evaluate an initial task strongly related to localization that consists in detecting whether two images have been captured in the same or in different rooms. Next, we assess Siamese Neural Networks in the context of a global localization problem. The results outperform previous techniques for solving the localization task using the COLD-Freiburg dataset, in a variety of lighting conditions, specially when using images captured in cloudy and night conditions.

Gated Siamese Fusion Network based on multimodal deep and hand-crafted features for personality traits assessment

People tend to judge others assessing their personality traits relying on life experience. This fact is especially evident when making an informed hiring decision, which should consider not only skills, but also match a company’s values and culture. Based on this assumption, we use the Siamese Network (SN) for assessing five personality traits by pairwise analyzing and comparing people simultaneously. For this, we propose the OCEAN-AI framework based on Gated Siamese Fusion Network (GSFN), which comprises six modules and enables the fusion of hand-crafted and deep features across three modalities (video, audio, and text). We use the ChaLearn First Impressions v2 (FIv2) and Multimodal Personality Traits Assessment (MuPTA) corpora and identify that all six feature sets and their combinations due to different information content allow the framework to adjust to heterogeneous input data flexibly. The experimental results show that the pairwise comparison of people with the same or different Personality Traits (PT) during the training enhances the proposed framework performance. The framework outperforms the State-of-the-Art (SOTA) systems based on three modalities (video-face, audio and text) by the relative value of 1.3% (0.928 vs. 0.916) in terms of the mean accuracy (mACC) on the FIv2 corpus. We also outperform the SOTA system in terms of the Concordance Correlation Coefficient (CCC) by the relative value of 8.6% (0.667 vs. 0.614) using two modalities (video and audio) on the MuPTA corpus. We make our framework publicly available to integrate it into various applications such as recruitment, education, and healthcare.

A deep learning-driven fingerprint verification model for enhancing exam integrity in Moroccan higher education

ABSTRACT In Moroccan higher education, the integrity of examinations is paramount, yet it faces the persistent challenge of identity impersonation. This form of academic dishonesty not only undermines the credibility of educational institutions but also contributes to the graduation of incompetent students. In an era where artificial intelligence is revolutionizing various sectors, its application to upholding academic integrity is both timely and essential. This paper proposes an advanced deep learning-driven fingerprint verification model specifically designed to combat impersonation in university examinations. Unlike traditional methods, our model leverages the power of Siamese neural networks (SNN), renowned for their effectiveness in learning distinct features and similarities. The model, trained, validated, and tested using the SOCOFing dataset, demonstrated high accuracy and effectiveness in fingerprint identification, crucial for verifying identities in educational exam settings. It achieved an accuracy of 99.29%, and an F1 score of 99.27%, surpassing other systems and significantly contributing to examination integrity in Moroccan higher education.

Hybridized soccer league-grey wolf Optimization algorithm and siamese network synergism based multimodal fusion scheme for prognosis of brain diseases

Hybridized soccer league-grey wolf Optimization algorithm and siamese network synergism based multimodal fusion scheme for prognosis of brain diseases

Enhanced classification of left ventricular hypertrophy in cardiac patients using extended Siamese CNN

Objective. Left ventricular hypertrophy (LVH) is the thickening of the left ventricle wall of the heart. The objective of this study is to develop a novel approach for the accurate assessment of LVH) severity, addressing the limitations of traditional manual grading systems. Approach. We propose the Multi-purpose Siamese Weighted Euclidean Distance Model (MSWED), which utilizes convolutional Siamese neural networks and zero-shot/few-shot learning techniques. Unlike traditional methods, our model introduces a cutoff distance-based approach for zero-shot learning, enhancing accuracy. We also incorporate a weighted Euclidean distance targeting informative regions within echocardiograms. Main results. We collected comprehensive datasets labeled by experienced echocardiographers, including Normal heart and various levels of LVH severity. Our model outperforms existing techniques, demonstrating significant precision enhancement, with improvements of up to 13% for zero-shot and few-shot learning approaches. Significance. Accurate assessment of LVH severity is crucial for clinical prognosis and treatment decisions. Our proposed MSWED model offers a more reliable and efficient solution compared to traditional grading systems, reducing subjectivity and errors while providing enhanced precision in severity classification.

Siamese Cooperative Learning for Unsupervised Image Reconstruction From Incomplete Measurements.

Image reconstruction from incomplete measurements is one basic task in imaging. While supervised deep learning has emerged as a powerful tool for image reconstruction in recent years, its applicability is limited by its prerequisite on a large number of latent images for model training. To extend the application of deep learning to the imaging tasks where acquisition of latent images is challenging, this article proposes an unsupervised deep learning method that trains a deep model for image reconstruction with the access limited to measurement data. We develop a Siamese network whose twin sub-networks perform reconstruction cooperatively on a pair of complementary spaces: the null space of the measurement matrix and the range space of its pseudo inverse. The Siamese network is trained by a self-supervised loss with three terms: a data consistency loss over available measurements in the range space, a data consistency loss between intermediate results in the null space, and a mutual consistency loss on the predictions of the twin sub-networks in the full space. The proposed method is applied to four imaging tasks from different applications, and extensive experiments have shown its advantages over existing unsupervised solutions.

Matching Patients to Clinical Trials using LLaMA 2 Embeddings and Siamese Neural Network.

Patient recruitment is a key desideratum for the success of a clinical trial that entails identifying eligible patients that match the selection criteria for the trial. However, the complexity of criteria information and heterogeneity of patient data render manual analysis a burdensome and time-consuming task. In an attempt to automate patient recruitment, this work proposes a Siamese Neural Network-based model, namely Siamese-PTM. Siamese-PTM employs the pretrained LLaMA 2 model to derive contextual representations of the EHR and criteria inputs and jointly encodes them using two weight-sharing identical subnetworks. We evaluate Siamese-PTM on structured and unstructured EHR to analyze their predictive informativeness as standalone and collective feature sets. We explore a variety of deep models for Siamese-PTM's encoders and compare their performance against the Single-encoder counterparts. We develop a baseline rule-based classifier, compared to which Siamese-PTM improved performance by 40%. Furthermore, visualization of Siamese-PTM's learned embedding space reinforces its predictive robustness.

TGC-ARG: Anticipating Antibiotic Resistance via Transformer-Based Modeling and Contrastive Learning.

In various domains, including everyday activities, agricultural practices, and medical treatments, the escalating challenge of antibiotic resistance poses a significant concern. Traditional approaches to studying antibiotic resistance genes (ARGs) often require substantial time and effort and are limited in accuracy. Moreover, the decentralized nature of existing data repositories complicates comprehensive analysis of antibiotic resistance gene sequences. In this study, we introduce a novel computational framework named TGC-ARG designed to predict potential ARGs. This framework takes protein sequences as input, utilizes SCRATCH-1D for protein secondary structure prediction, and employs feature extraction techniques to derive distinctive features from both sequence and structural data. Subsequently, a Siamese network is employed to foster a contrastive learning environment, enhancing the model's ability to effectively represent the data. Finally, a multi-layer perceptron (MLP) integrates and processes sequence embeddings alongside predicted secondary structure embeddings to forecast ARG presence. To evaluate our approach, we curated a pioneering open dataset termed ARSS (Antibiotic Resistance Sequence Statistics). Comprehensive comparative experiments demonstrate that our method surpasses current state-of-the-art methodologies. Additionally, through detailed case studies, we illustrate the efficacy of our approach in predicting potential ARGs.

Few-shot learning fault diagnosis of rolling bearings based on siamese network

Abstract This paper focuses on the fault diagnosis problem in the scenario of scarce bearing samples, facing two main challenges: complex noise background and variations in operating conditions. While deep learning-based fault diagnosis methods have achieved significant progress, they heavily rely on large amounts of samples. This paper proposes a few-shot learning fault diagnosis method based on siamese networks (SN), which classify samples based on the similarity between pairs rather than end-to-end classification. Tested on two bearing datasets, the proposed method outperforms SVM, DCNN, WDCNN, and CNN-BiGRU. The influence of factors such as parameter regularization, noise, and load variation on the proposed method is also discussed. Experimental results demonstrate that double parameter regularization contributes more to the model’s generalization ability, maintaining good stability and generalization even under noise interference or load variation.

AISMSNet: Advanced Image Splicing Manipulation Identification Based on Siamese Networks

The exponential surge in specialized image editing software has intensified visual forgery, with splicing attacks emerging as a popular forgery technique. In this context, Siamese neural networks are a remarkable tool in pattern identification for detecting image manipulations. This paper introduces a deep learning approach for splicing detection based on a Siamese neural network tailored to identifying manipulated image regions. The Siamese neural network learns unique features of specific image areas and detects tampered regions through feature comparison. This architecture employs two identical branches with shared weights and image features to compare image blocks and identify tampered areas. Subsequently, a K-means algorithm is applied to identify similar centroids and determine the precise localization of duplicated regions in the image. The experimental results encompass various splicing attacks to assess effectiveness, demonstrating a high accuracy of 98.6% and a precision of 97.5% for splicing manipulation detection. This study presents an advanced splicing image forgery detection and localization algorithm, showcasing its efficacy through comprehensive experiments.

Deep learning fusion for dynamic cropland monitoring: integrating MUNet with Siamese networks and convolutional LSTM2D

Deep learning fusion for dynamic cropland monitoring: integrating MUNet with Siamese networks and convolutional LSTM2D

Identifying radiogenomic associations of breast cancer based on DCE-MRI by using Siamese Neural Network with manufacturer bias normalization.

The immunohistochemical test (IHC) for Human Epidermal Growth Factor Receptor 2 (HER2) and hormone receptors (HR) provides prognostic information and guides treatment for patients with invasive breast cancer. The objective of this paper is to establish a non-invasive system for identifying HER2 and HR in breast cancer using dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI). In light of the absence of high-performance algorithms and external validation in previously published methods, this study utilizes 3D deep features and radiomics features to represent the information of the Region of Interest (ROI). A Siamese Neural Network was employed as the classifier, with 3D deep features and radiomics features serving as the network input. To neutralize manufacturer bias, a batch effect normalization method, ComBat, was introduced. To enhance the reliability of the study, two datasets, Predict Your Therapeutic Response with Imaging and moLecular Analysis (I-SPY 1) and I-SPY 2, were incorporated. I-SPY 2 was utilized for model training and validation, while I-SPY 1 was exclusively employed for external validation. Additionally, a breast tumor segmentation network was trained to improve radiomic feature extraction. The results indicate that our approach achieved an average Area Under the Curve (AUC) of 0.632, with a Standard Error of the Mean (SEM) of 0.042 for HER2 prediction in the I-SPY 2 dataset. For HR prediction, our method attained an AUC of 0.635 (SEM 0.041), surpassing other published methods in the AUC metric. Moreover, the proposed method yielded competitive results in other metrics. In external validation using the I-SPY 1 dataset, our approach achieved an AUC of 0.567 (SEM 0.032) for HR prediction and 0.563 (SEM 0.033) for HER2 prediction. This study proposes a non-invasive system for identifying HER2 and HR in breast cancer. Although the results do not conclusively demonstrate superiority in both tasks, they indicate that the proposed method achieved good performance and is a competitive classifier compared to other reference methods. Ablation studies demonstrate that both radiomics features and deep features for the Siamese Neural Network are beneficial for the model. The introduced manufacturer bias normalization method has been shown to enhance the method's performance. Furthermore, the external validation of the method enhances the reliability of this research. Source code, pre-trained segmentation network, Radiomics and deep features, data for statistical analysis, and Supporting Information of this article are online at: https://github.com/FORRESTHUACHEN/Siamese_Neural_Network_based_Brest_cancer_Radiogenomic.