Siamese Network Research Articles

Augmented histopathology: Enhancing colon cancer detection through deep learning and ensemble techniques.

Colon cancer poses a significant threat to human life with a high global mortality rate. Early and accurate detection is crucial for improving treatment quality and the survival rate. This paper presents a comprehensive approach to enhance colon cancer detection and classification. The histopathological images are gathered from the CRC-VAL-HE-7K dataset. The images undergo preprocessing to improve quality, followed by augmentation to increase dataset size and enhance model generalization. A deep learning based transformer model is designed for efficient feature extraction and enhancing classification by incorporating a convolutional neural network (CNN). A cross-transformation model captures long-range dependencies between regions, and an attention mechanism assigns weights to highlight crucial features. To boost classification accuracy, a Siamese network distinguishes colon cancer tissue classes based on probabilities. Optimization algorithms fine-tune model parameters, categorizing colon cancer tissues into different classes. The multi-class classification performance is evaluated in the experimental evaluation, which demonstrates that the proposed model provided highest accuracy rate of 98.84%. In this research article, the proposed method achieved better performance in all analyses by comparing with other existing methods. RESEARCH HIGHLIGHTS: Deep learning-based techniques are proposed. DL methods are used to enhance colon cancer detection and classification. CRC-VAL-HE-7K dataset is utilized to enhance image quality. Hybrid particle swarm optimization (PSO) and dwarf mongoose optimization (DMO) are used. The deep learning models are tuned by implementing the PSO-DMO algorithm.

SiamATA: an asymmetric target-aware and frequency domain task-aware Siamese network for visual tracking

SiamATA: an asymmetric target-aware and frequency domain task-aware Siamese network for visual tracking

IMCSN: An improved neighborhood aggregation interaction strategy for multi-scale contrastive Siamese networks

Graph contrastive learning is an effective method for enhancing graph representation by maximizing the similarity of representations between analogous graphs while minimizing the similarity between dissimilar ones. A common approach to improve the representation capabilities of graphs involves data augmentation, which generates additional training samples through transformations applied to the original graphs. However, traditional data augmentation techniques, such as random deletion of nodes or edges, often compromise the structural integrity of graphs, result in loss of crucial information, and lead to representation instability. To overcome these drawbacks, this study introduces a novel data augmentation paradigm, the integration of directed random noise (IDR), which incorporates controlled random noise to optimize the augmentation objectives. IDR enhances the diversity and robustness of representations without sacrificing the structural integrity of the graphs. This approach significantly improves the performance of graph contrastive learning, avoiding the common issues of graph structure damage and information loss associated with conventional methods. To further refine the model, this paper proposes an improved multiscale contrastive Siamese network framework that employs three Siamese networks to process different views of input graphs. It utilizes cross-network and cross-view contrastive learning objectives to optimize graph representations, leveraging the complementary information between different views to maximize the consistency of representations among similar graphs. This enhancement improves the quality and generalizability of graph representations. Additionally, the introduction of a self-supervised loss function based on graph reconstruction within the loss framework capitalizes on the structural similarity between the original and reconstructed graphs to further refine graph representations. This loss function ensures that the global view retains more structural information from the original graph, thus enhancing the complementarity between the global and local views. The effectiveness and stability of this research framework are demonstrated through node classification tasks and visualizations on six real-world datasets, comparing favorably against existing methods such as MERIT and GraphVAT. The proposed model achieves accuracy improvements of 3.21 % and 3.71 % on the Cora dataset and 1.12 % and 1.42 % on the CiteSeer dataset. Additionally, it ranks first in an average accuracy comparison across six datasets, with scores of 80.6 % and 52.67 %. These results underscore the robustness and efficacy of the proposed model in self-supervised graph representation learning, offering substantial advancements over existing techniques.

Multimodal masked siamese network improves chest X-ray representation learning

Self-supervised learning methods for medical images primarily rely on the imaging modality during pretraining. Although such approaches deliver promising results, they do not take advantage of the associated patient or scan information collected within Electronic Health Records (EHR). This study aims to develop a multimodal pretraining approach for chest radiographs that considers EHR data incorporation as an additional modality that during training. We propose to incorporate EHR data during self-supervised pretraining with a Masked Siamese Network (MSN) to enhance the quality of chest radiograph representations. We investigate three types of EHR data, including demographic, scan metadata, and inpatient stay information. We evaluate the multimodal MSN on three publicly available chest X-ray datasets, MIMIC-CXR, CheXpert, and NIH-14, using two vision transformer (ViT) backbones, specifically ViT-Tiny and ViT-Small. In assessing the quality of the representations through linear evaluation, our proposed method demonstrates significant improvement compared to vanilla MSN and state-of-the-art self-supervised learning baselines. In particular, our proposed method achieves an improvement of of 2% in the Area Under the Receiver Operating Characteristic Curve (AUROC) compared to vanilla MSN and 5% to 8% compared to other baselines, including uni-modal ones. Furthermore, our findings reveal that demographic features provide the most significant performance improvement. Our work highlights the potential of EHR-enhanced self-supervised pretraining for medical imaging and opens opportunities for future research to address limitations in existing representation learning methods for other medical imaging modalities, such as neuro-, ophthalmic, and sonar imaging.

Small object change detection in UAV imagery via a Siamese network enhanced with temporal mutual attention and contextual features: A case study concerning solar water heaters

Small object change detection (SOCD) based on high-spatial resolution (HSR) images is of significant practical value in applications such as the investigation of illegal urban construction, but little research is currently available. This study proposes an SOCD model called TMACNet based on a multitask network architecture. The model modifies the YOLOv8 network into a Siamese network and adds structures, including a feature difference branch (FDB), temporal mutual attention layer (TMAL) and contextual attention module (CAM), to merge differential and contextual features from different phases for the accurate extraction and analysis of small objects and their changes. To verify the proposed method, an SOCD dataset called YZDS is created based on unmanned aerial vehicle (UAV) images of small-scale solar water heaters on rooftops. The experimental results show that TMACNet exhibits strong resistance to image registration errors and building height displacement and prevents error propagation from object detection to change detection originating from overlay-based change detection. TMACNet also provides an enhanced approach to small object detection from the perspective of multitemporal information fusion. In the change detection task, TMACNet exhibits notable F1 improvements exceeding 5.96% in comparison with alternative change detection methods. In the object detection task, TMACNet outperforms the single-temporal object detection models, increasing accuracy with an approximately 1–3% improvement in the AP metric while simplifying the technical process.

A Scene Graph Similarity-Based Remote Sensing Image Retrieval Algorithm

With the rapid development of remote sensing image data, the efficient retrieval of target images of interest has become an important issue in various applications including computer vision and remote sensing. This research addressed the low-accuracy problem in traditional content-based image retrieval algorithms, which largely rely on comparing entire image features without capturing sufficient semantic information. We proposed a scene graph similarity-based remote sensing image retrieval algorithm. Firstly, a one-shot object detection algorithm was designed for remote sensing images based on Siamese networks and tailored to the objects of an unknown class in the query image. Secondly, a scene graph construction algorithm was developed, based on the objects and their attributes and spatial relationships. Several construction strategies were designed based on different relationships, including full connections, random connections, nearest connections, star connections, or ring connections. Thirdly, by making full use of edge features for scene graph feature extraction, a graph feature extraction network was established based on edge features. Fourthly, a neural tensor network-based similarity calculation algorithm was designed for graph feature vectors to obtain image retrieval results. Fifthly, a dataset named remote sensing images with scene graphs (RSSG) was built for testing, which contained 929 remote sensing images with their corresponding scene graphs generated by the developed construction strategies. Finally, through performance comparison experiments with remote sensing image retrieval algorithms AMFMN, MiLaN, and AHCL, in precision rates, Precision@1 improved by 10%, 7.2%, and 5.2%, Precision@5 improved by 3%, 5%, and 1.7%; and Precision@10 improved by 1.7%, 3%, and 0.6%. In recall rates, Recall@1 improved by 2.5%, 4.3%, and 1.3%; Recall@5 improved by 3.7%, 6.2%, and 2.1%; and Recall@10 improved by 4.4%, 7.7% and 1.6%.

Extremely compact video representation for efficient near-duplicates detection

With the influx of online video uploads, a robust method for detecting video copies under various distortions is essential for copyright protection and search efficiency. We propose a compact video representation that uses a Siamese Neural Network for near-duplicate detection. This approach achieves up to 0.998 recall and 0.853 precision, even with distorted 56x56px miniatures. Consequently, we derive extremely compact video descriptors, facilitating video fragment retrieval in large datasets. Our method pre-selects frames from a video by identifying local maximums from the interframe differences curve, preserving characteristic sequence patterns even after extreme compression. Utilizing a simpler convolution-based model in the Siamese Neural Network, we improve results by up to 8% over VGG-16, with a 1.7-fold reduction in inference time. With compact descriptors of 1.875 kB, our models outperform others by more than two times. Additionally, we introduce a new dataset with dynamically changing scenes, enhancing its suitability for near-duplicate video detection.

Fusion of Multiple Attention Mechanisms and Background Feature Adaptive Update Strategies in Siamese Networks for Single-Object Tracking

Single-object tracking algorithms based on Siamese full convolutional networks have attracted much attention from researchers owing to their improvement in precision and speed. Since this tracking model only learns a similarity model offline, it is not able to obtain more useful feature discrimination information to adapt to the various variations of targets in complex scenes. To improve the performance of this tracking model, we propose a Siamese network tracking algorithm that incorporates multiple attention mechanisms and an adaptive updating strategy for background features. First, a backbone feature extraction network is proposed that utilizes a small convolutional kernel to fuse jump-layer connectivity features, thereby improving the feature representation capability of the network. Second, an adaptive update strategy for background features is proposed to improve the model’s ability to discriminate between the object and background features. Third, the fusion of multiple attention mechanisms is proposed so that the model learns to focus on the channel, spatial, and coordinate features. Fourth, the response fusion operation is proposed after the inter-correlation operation to enrich the output response of the model. Finally, our algorithm is trained using the GOT-10K dataset and evaluated by testing on the object tracking benchmark datasets OTB100 and VOT2018. The test results show that compared with other algorithms, our algorithm can effectively cope with the problem of degradation of the tracking performance in complex environments, and it can further improve the tracking precision and precision under the premise of ensuring the tracking speed.

Exploring Siamese network to estimate sea state bias of synthetic aperture radar altimeter

Sea state bias (SSB) is a crucial error of satellite radar altimetry over the ocean surface. For operational nonparametric SSB (NPSSB) models, such as two-dimensional (2D) or three-dimensional (3D) NPSSB, the solution process becomes increasingly complex and the construction of their regression functions pose challenges as the dimensionality of relevant variables increases. And most current SSB correction models for altimeters still follow those of traditional nadir radar altimeters, which limits their applicability to Synthetic Aperture Radar altimeters. Therefore, to improve this situation, this study has explored the influence of multi-dimensional SSB models on Synthetic Aperture Radar altimeters. This paper proposes a deep learning-based SSB estimation model called SNSSB, which employs a Siamese network framework, takes various multi-dimensional variables related to sea state as inputs, and uses the difference in sea surface height (SSH) at self-crossover points as the label. Experiments were conducted using Sentinel-6 self-crossover data from 2021 to 2023, and the model is evaluated using three main metrics: the variance of the SSH difference, the explained variance, and the SSH difference variance index (SVDI). The experimental results demonstrate that the proposed SNSSB model can further improve the accuracy of SSB estimation. On a global scale, compared to the traditional NPSSB, the multi-dimensional SNSSB not only decreases the variance of the SSH difference by over 11%, but also improves the explained variance by 5-10 cm2 in mid- and low-latitude regions. And the regional SNSSB also performs well, reducing the variance of the SSH difference by over 10% compared to the NPSSB. Additionally, the SNSSB model improves the computational efficiency by approximately 100 times. The favorable results highlight the potential of the multi-dimensional SNSSB in constructing SSB models, particularly the five-dimensional (5D) SNSSB, representing a breakthrough in overcoming the limitations of traditional NPSSB for constructing high-dimensional models. This study provides a novel approach to exploring the multiple influencing factors of SSB.

Object matching of visible–infrared image based on attention mechanism and feature fusion

Object matching can be viewed as an image patch matching problem, which is widely employed in image fusion, image retrieval and other computer vision fields. In this paper, we regard image matching as a regression and classification task and propose a visible and infrared image matching network named AMFFNet. AMFFNet adopts a Siamese network structure and utilizes a residual network with an attention mechanism to extract features from the input images to obtain feature maps, and fuses these feature maps. AMFFNet then performs classification and regression on the fused feature maps to achieve matching. The classification operation identifies whether the predicted area is the object, while the regression operation determines the coordinates and size of the predicted box. To improve the matching performance of the network, we utilize the center-ness branch in the network and use Generalized Intersection over Union (GIoU) loss during training. We rearrange the existing dataset to provide a sufficient visible–infrared image matching dataset. Experimental results demonstrate that the proposed method achieves superior matching mean Average Precision (mAP) compared to other methods.

Decoupling foreground and background with Siamese ViT networks for weakly-supervised semantic segmentation

Due to the coarse granularity of information extraction in image-level annotation-based weakly supervised semantic segmentation algorithms, there exists a significant gap between the generated pseudo-labels and the real pixel-level labels. In this paper, we propose the DeFB-SV framework, which consists of a dual-branch Siamese network structure. This framework separates the foreground and background of images by generating unified resolution and mixed resolution class activation maps, which are then fused to obtain pseudo-labels. The mixed-resolution class activation maps are produced by a new mixed-resolution patch partition method, where we introduce a semantically heuristic patch scorer to divide the image into patches of different sizes based on semantics. Additionally, a novel multi-confidence region division mechanism is proposed to enable the adaptive extraction of the effective parts of pseudo-labels, further enhancing the accuracy of weakly supervised semantic segmentation algorithms. The proposed semantic segmentation framework, DeFB-SV, is evaluated on the PASCAL VOC 2012 and MS COCO 2014 datasets, demonstrating comparable segmentation performance with state-of-the-art methods.

Sia-m7G: Predicting m7G Sites through the Siamese Neural Network with an Attention Mechanism

Background: The chemical modification of RNA plays a crucial role in many biological processes. N7-methylguanosine (m7G), being one of the most important epigenetic modifications, plays an important role in gene expression, processing metabolism, and protein synthesis. Detecting the exact location of m7G sites in the transcriptome is key to understanding their relevant mechanism in gene expression. On the basis of experimentally validated data, several machine learning or deep learning tools have been designed to identify internal m7G sites and have shown advantages over traditional experimental methods in terms of speed, cost-effectiveness and robustness. Aims: In this study, we aim to develop a computational model to help predict the exact location of m7G sites in humans. Objective: Simple and advanced encoding methods and deep learning networks are designed to achieve excellent m7G prediction efficiently. Methods: Three types of feature extractions and six classification algorithms were tested to identify m7G sites. Our final model, named Sia-m7G, adopts one-hot encoding and a delicate Siamese neural network with an attention mechanism. In addition, multiple 10-fold cross-validation tests were conducted to evaluate our predictor. Results: Sia-m7G achieved the highest sensitivity, specificity and accuracy on 10-fold crossvalidation tests compared with the other six m7G predictors. Nucleotide preference and model visualization analyses were conducted to strengthen the interpretability of Sia-m7G and provide a further understanding of m7G site fragments in genomic sequences. Conclusion: Sia-m7G has significant advantages over other classifiers and predictors, which proves the superiority of the Siamese neural network algorithm in identifying m7G sites.

SiaDFP: A Disk Failure Prediction Framework Based on Siamese Neural Network in Large-Scale Data Center

SiaDFP: A Disk Failure Prediction Framework Based on Siamese Neural Network in Large-Scale Data Center

Relative position matrix and multi-scale feature fusion for writer-independent online signature verification

Online signature verification (OSV) is widely used in finance, law and other fields, and is one of the important research projects on biological characteristics. However, its data set has a small scale and has high requirements for generalization of certification models. Therefore, how to overcome these problems is of great value to improve the practicality and security of online handwriting signature technology. We propose a writer-independent online handwritten signature verification method, which adopts the relative position matrix method to convert the traditional temporal features into images for processing. This method enriched the features of the signatures, serving the purpose of data augmentation. Then two-dimensional multi-scale feature fusion based Siamese neural network (2D-MFFnet) is built for representing and learning the importance of each channel adaptively combined with the attention mechanism. Finally, a temporal convolutional network is designed to construct the classifier. The results illustrate that compared with traditional time series models, the algorithm has reduced the equal error rate by at least 2.52 % on the open datasets MCYT-100 and SVC2004 task2.

6-Degree Vision based Tracking of a Mandible Phantom with Deep Learning

Abstract During maxillofacial surgery, the precise placement of surgical tools is crucial for accurate implant placement. Particularly if multiple implants are needed, e.g., after cancerous bone removal, visual landmarks might not be obtainable. To this end, we propose a vision-based tracking approach with deep learning. Our markerless tracking approach is based on video streams from two cameras for tracking a mandible phantom. We study, to what extent vision-based localization using deep learning is feasible. For real-time 6D pose estimation we propose a Siamese network with a ResNet-18 subnetwork. We acquire a large training dataset with a robot and evaluate the tracking accuracy on partially occluded images. Thereby, we mimic visual information that is accessible during clinical interventions. We report a mean position error of 1.33 ± 1.14mm and a rotation error of 0.86 ± 0.71 deg for partially occluded images. Overall we present a promising tracking approach that is marker-free and robust toward image artifacts.

Siamese network for classification of Raman spectroscopy with inter-instrument variation for biological applications

Raman spectroscopy has emerged as a highly sensitive, rapid, and label-free detection method, extensively utilized in biological research. Presently, it is frequently paired with artificial intelligence (AI) algorithms to facilitate identification and classification tasks. However, variations in the settings across different Raman spectrometers, along with the sensitive and continuous nature of biological Raman signals, can subtly alter the acquisition of these signals. This can potentially impact the classification outcomes of the spectra. Moreover, Raman spectra with disparate resolutions pose challenges for effective model training. In this study, we introduce a modularized Siamese neural network, equipped with multiple projection layers to segregate the model components. This design allows our model to support the core module spectral encoder’s pluggability. The model determines the classification results by extracting the features of Raman spectra with inter-instrument variation, mapping these feature distances into spectral similarities, and finally, comparing a set of similarities. Our experimental results demonstrate the feasibility of training the model with only 10 spectra per category, using bacterial datasets we created. We compared the classification outcomes of three distinct spectral encoders, with the most effective model achieving a classification accuracy exceeding 90%. Furthermore, we successfully implemented the fusion training and prediction of Raman spectra with different resolutions. In conclusion, our model enhances the validity and comparability of Raman spectral acquisition for biological applications and diversifies the methods of Raman spectral acquisition.

A Generalisable Heartbeat Classifier Leveraging Self-Supervised Learning for ECG Analysis During Magnetic Resonance Imaging.

Electrocardiogram (ECG) is acquired during Magnetic Resonance Imaging (MRI) to monitor patients and synchronize image acquisition with the heart motion. ECG signals are highly distorted during MRI due to the complex electromagnetic environment. Automated ECG analysis is therefore complicated in this context and there is no reference technique in MRI to classify pathological heartbeats. Imaging arrhythmic patients is hence difficult in MRI. Deep Learning based heartbeat classifier have been suggested but require large databases whereas existing annotated sets of ECG in MRI are very small. We proposed a Siamese network to leverage a large database of unannotated ECGs outside MRI. This was used to develop an efficient representation of ECG signals, further used to develop a heartbeat classifier. We extensively tested several data augmentations and self-supervised learning (SSL) techniques and assessed the generalization of the obtained classifier to ECG signals acquired in MRI. These augmentations included random noises and a model simulating MRI specific artefacts. SSL pretraining improved the generalizability of heartbeat classifiers in MRI (F1=0.75) compared to Deep Learning not relying on SSL (F1=0.46) and another classical machine learning approach (F1=0.40). These promising results seem to indicate that the use of SSL techniques can learn efficient ECG signal representation, and are useful for the development of Deep Learning models even when only scarce annotated medical data are available.

Deep learning-based authentication for insider threat detection in critical infrastructure

In today’s cyber environment, threats such as data breaches, cyberattacks, and unauthorized access threaten national security, critical infrastructure, and financial stability. This research addresses the challenging task of protecting critical infrastructure from insider threats because of the high level of trust and access these individuals typically receive. Insiders may obtain a system administrator’s password through close observation or by deploying software to gather the information. To solve this issue, an innovative artificial intelligence-based methodology is proposed to identify a user by their password’s keystroke dynamics. This paper also introduces a new Gabor Filter Matrix Transformation method to transform numerical values into images by revealing the behavioral pattern of password typing. A siamese neural network (SNN) with the branches of convolutional neural networks is utilized for image comparison, aiming to detect unauthorized attempts to access critical infrastructure systems. The network analyzes the unique features of a user’s password timestamps transformed into images and compares them with previously submitted user passwords. The obtained results indicate that transforming the numerical values of keystroke dynamics into images and training an SNN leads to a lower equal error rate (EER) and higher user authentication accuracy than those previously reported in other studies. The methodology is validated on publicly available keystroke dynamics collections, the CMU and GREYC-NISLAB datasets, which collectively comprise over 30,000 password samples. It achieves the lowest EER value of 0.04545 compared to state-of-the-art methods for transforming non-image data into images. The paper concludes with a discussion of findings and potential future directions.

Long-term time series forecasting based on Siamese network: a perspective on few-shot learning

Long-term time series forecasting based on Siamese network: a perspective on few-shot learning

Symmetrical Siamese Network for pose-guided person synthesis

Pose-Guided Person Image Synthesis (PGPIS) aims to generate a realistic person image that preserves the appearance of the source person while adopting the target pose. Various appearances and drastic pose changes make this task highly challenging. Due to the insufficient utilization of paired data, existing models face difficulties in accurately preserving the source appearance details and high-frequency textures in the generated images. Meanwhile, although current popular AdaIN-based methods are advantageous in handling drastic pose changes, they struggle to capture diverse clothing shapes imposed by the limitation of global feature statistics. To address these issues, we propose a novel Symmetrical Siamese Network (SSNet) for PGPIS, which consists of two synergistic symmetrical generative branches that leverage prior knowledge of paired data to comprehensively exploit appearance details. For feature integration, we propose a Style Matching Module (SMM) to transfer multi-level region appearance styles and gradient information to the desired pose for enriching the high-frequency textures. Furthermore, to overcome the limitation of global feature statistics, a Spatial Attention Module (SAM) is introduced to complement the SMM for capturing clothing shapes. Extensive experiments show the effectiveness of our SSNet, achieving state-of-the-art results on public datasets. Moreover, our SSNet can also edit the source appearance attributes, making it versatile in wider application scenarios.