Image Modalities Research Articles

OBIA: An Open Biomedical Imaging Archive

With the development of artificial intelligence (AI) technologies, biomedical imaging data play an important role in scientific research and clinical application, but the available resources are limited. Here we present Open Biomedical Imaging Archive (OBIA), a repository for archiving biomedical imaging and related clinical data. OBIA adopts five data objects (Collection, Individual, Study, Series, and Image) for data organization, and accepts the submission of biomedical images of multiple modalities, organs, and diseases. In order to protect personal privacy, OBIA has formulated a unified de-identification and quality control process. In addition, OBIA provides friendly and intuitive web interfaces for data submission, browsing, and retrieval, as well as image retrieval. As of September 2023, OBIA has housed data for a total of 937 individuals, 4136 studies, 24,701 series, and 1,938,309 images covering 9 modalities and 30 anatomical sites. Collectively, OBIA provides a reliable platform for biomedical imaging data management and offers free open access to all publicly available data to support research activities throughout the world. OBIA can be accessed at https://ngdc.cncb.ac.cn/obia.

Infrared and visible image fusion via mixed-frequency hierarchical guided learning

Existing deep learning-based fusion methods usually model local information through convolution operation or global contexts via self-attention mechanism. This serial scheme discards the local features during globality modeling, or vice versa, which may generate limited fusion performance. To tackle this issue, we introduce a mixed-frequency hierarchical guided learning network, or FreqFuse for short. More specifically, we first design a parallel frequency mixer through a channel splitting mechanism, including max-pooling and self-attention paths, to learn both high and low-frequency information. The mixer can provide more comprehensive features within a wide frequency range compared with a single dependency. Second, we develop a dual-Transformer integration module to guide fusion progress. The assigned weights are calculated by cross-token and cross-channel Transformer, which are used to measure the activity levels of source images and preserve their modality characteristics in the intermediate fused features. On this basis, we build a hierarchical guidance decoder to reconstruct a final fusion image. The cross-scale mixed-frequency features are reused to gradually optimize the activity levels of different modality images, and promote the fused result to be highly informative and strongly characterized. We benchmark the proposed FreqFuse on different datasets, and experimental results demonstrate that it achieves impressive performance compared with other methods.

Clinical Usability-Oriented Automatic Contour Quality Evaluation for Deep Learning Auto-Segmentation

Various auto-segmentations, including deep learning auto-segmentation (DLAS), are being increasingly adopted in radiotherapy. A common method to evaluate quality of auto-segmented contours uses thresholds of various quantitative metrics (e.g., dice similarity coefficient (DSC), mean distance to agreement (MDA), etc.) that are often averaged over all contour slices. This method fails to detect contour errors on individual slices, thus, does not reflect the current clinical practice (slice-by-slice evaluation) and the clinical usability (e.g., expected contour editing time). In addition, the use of multi-metrics is generally not easy to interpret. This work aims to develop a novel contour quality classification (CQC) model to evaluate auto-segmented contours based on their clinical applicability. The CQC method was designed to classify a contour on a slice into acceptable, minor edit or major edit category, based on the expected editing effort/time. Organ-specific supervised ensemble tree classification models were trained to relate the slice-based quality category with the combination of seven commonly used calculatable quantitative metrics (i.e., DSC, MDA, Hausdorff 95% distance, surface DSC, added path length (APL), slice area and relative APL). The proposed method was demonstrated by training CQC models using DLAS contours of five abdominal organs (i.e., pancreas, duodenum, stomach, and small and large bowels) from 50 MRI sets and evaluating on 20 MRI and 9 CT testing sets. These test datasets were labelled by six individual observers and the consensus labels were generated through majority vote method. The model performance was evaluated using accuracy (acc), and risk rate (RR, the percentage of unacceptable slices mislabeled as acceptable) and compared with inter-observer variation and baseline threshold-based method. Compared to the majority vote labels, the obtained CQC models achieved a mean accuracy of 95.8% ([94.5%-99.1%]) and 94.3% ([90.6%-96.9%]), and the mean RR of 0.8% ([0.3%-1.3%]) and 0.7% ([0%-1.1%]) for the MRI and CT testing sets, respectively. The CQC performance was comparable to the inter-observer variation and significantly higher than those from the threshold-based method with single or multiple metrics. The execution time on a typical abdominal dataset (e.g., 70 slices) took less than 3 seconds. Table 1 CQC models performance for different organs CONCLUSION: The proposed CQC model can classify the quality of a contour slice with high accuracy. This slice-based single-output evaluation method better reflects the current clinical practice and may be used to evaluate/compare performance of DLAS on any image modality, facilitating its clinical implementation and quality assurance.

Dual-modality image feature fusion network for gastric precancerous lesions classification

Gastric precancerous conditions are closely linked to the development of gastric cancer. However, the detection of gastric precancerous lesions (GPL) is limited by the indistinct symptoms and the low detection rate of microscope images. This paper proposes an RGB and Hyperspectral Dual-modality imaging Feature Fusion Network (DuFF-Net) to improve the classification accuracy of GPL. To fully exploit information of different modality images, we customize a dual-stream ResNet-based model for feature sharing and fusion. Skip-Connections are added between inter-path of networks to achieve information interaction. In the decision step, we adopt the SE-based attention module and Pearson Correlation to highlight and select effective features. Experimental results show that the DuFF-Net increases the screening accuracy to 96.15 % for two types of gastric precancerous tissues with high morphological similarity. Furthermore, our approach reduces the labeling workload for classification tasks by approximately 50 %. These findings provide valuable guidance for the screening and subsequent lesion segmentation of GPLs.

Clinical Feasibility of Deep Learning-Based CT during Treatment CBCT Tumor Registration-Segmentation in Thoracic Radiotherapy (RT)

Accurate tumor segmentation on weekly cone-beam computed tomography (CBCT) images is critical for image-guided and adaptive radiation therapy (ART). In thoracic RT, low image contrast, imaging artifact, and geometry and image modality differences from the planning CT (pCT) typically limits accurate tumor segmentation and registration. Here, we explored the clinical feasibility of using 3D recurrent registration-segmentation deep learning (DL) that combines patient-specific anatomic and shape context from higher contrast pCT and planning contours (PACs) for tumor segmentation on during treatment CBCTs. We included the pCT and CBCTs from six patients with locally advanced non-small cell lung cancer (LA-NSCLC) who had underwent RT. Cases were selected with a primary GTV contoured and labeled separately from the nodal GTV. Using rigidly aligned pCT and CBCT as inputs, DL auto-segmented the GTV on week 1 and 6 CBCTs, and these auto-segmented contours were manually inspected by a radiation oncologist that edited the GTV according to clinical standard quality. The Dice similarity coefficient (DSC), Hausdorff distance (HD95), mean surface distance (MSD), surface DSC (sDSC) and added path length (APL) were used to quantitively compare the DL and the edited GTV. The primary GTV was in the right lung in five cases, and left lung in one case. Manual adjustments were typically made at the interface of GTV and lung parenchyma with partial inclusion of adjacent vessels. Hypodensities within the GTV were sometimes not segmented in all axial slices resulting in discontinuous components. The quantitative comparison between the edited and DL-generated GTV is shown in Table 1. For week 1, the average DSC and HD95 were 0.87 and 6.94 mm, respectively. The performance for week 6 was slightly lower than week 1, with a DSC of 0.85 and HD95 of 7.22 mm. The agreement with the generated DL GTV and the edited GTV was high in week 1 and decreased somewhat later during the treatment course possibly due to a higher impact of geometric changes in tumor and adjacent structures. The proposed DL algorithm showed reasonable performance throughout the treatment, supporting its potential for use into clinical routine for LA-NSCLC.

A systematic review of machine and deep learning techniques for the identification and classification of breast cancer through medical image modalities

A systematic review of machine and deep learning techniques for the identification and classification of breast cancer through medical image modalities

6D Object Pose Estimation Based on Cross-Modality Feature Fusion.

The 6D pose estimation using RGBD images plays a pivotal role in robotics applications. At present, after obtaining the RGB and depth modality information, most methods directly concatenate them without considering information interactions. This leads to the low accuracy of 6D pose estimation in occlusion and illumination changes. To solve this problem, we propose a new method to fuse RGB and depth modality features. Our method effectively uses individual information contained within each RGBD image modality and fully integrates cross-modality interactive information. Specifically, we transform depth images into point clouds, applying the PointNet++ network to extract point cloud features; RGB image features are extracted by CNNs and attention mechanisms are added to obtain context information within the single modality; then, we propose a cross-modality feature fusion module (CFFM) to obtain the cross-modality information, and introduce a feature contribution weight training module (CWTM) to allocate the different contributions of the two modalities to the target task. Finally, the result of 6D object pose estimation is obtained by the final cross-modality fusion feature. By enabling information interactions within and between modalities, the integration of the two modalities is maximized. Furthermore, considering the contribution of each modality enhances the overall robustness of the model. Our experiments indicate that the accuracy rate of our method on the LineMOD dataset can reach 96.9%, on average, using the ADD (-S) metric, while on the YCB-Video dataset, it can reach 94.7% using the ADD-S AUC metric and 96.5% using the ADD-S score (<2 cm) metric.

Imaging of excised cochleae by micro-CT: staining, liquid embedding, and image modalities.

Assessing the complex three-dimensional (3D) structure of the cochlea is crucial to understanding the fundamental aspects of signal transduction in the inner ear and is a prerequisite for the development of novel cochlear implants. X-ray phase-contrast computed tomography offers destruction-free 3D imaging with little sample preparation, thus preserving the delicate structure of the cochlea. The use of heavy metal stains enables higher contrast and resolution and facilitates segmentation of the cochlea. For μ-CT of small animal and human cochlea, we explore the heavy metal osmium tetroxide (OTO) as a radiocontrast agent and delineate laboratory from synchrotron CT. We investigate how phase retrieval can be used to improve the image quality of the reconstructions, both for stained and unstained specimens. Image contrast for soft tissue in an aqueous solution is insufficient under the in-house conditions, whereas the OTO stain increases contrast for lipid-rich tissue components, such as the myelin sheaths in nervous tissue, enabling contrast-based rendering of the different components of the auditory nervous system. The overall morphology of the cochlea with the three scalae and membranes is very well represented. Further, the image quality of the reconstructions improves significantly when a phase retrieval scheme is used, which is also suitable for non-ideal laboratory settings. With highly brilliant synchrotron radiation (SR), we achieve high contrast for unstained whole cochleae at the cellular level. The OTO stain is suitable for 3D imaging of small animal and human cochlea with laboratory , and relevant pathologies, such as a loss of sensory cells and neurons, can be visualized. With SR and optimized phase retrieval, the cellular level can be reached even for unstained samples in aqueous solution, as demonstrated by the high visibility of single hair cells and spiral ganglion neurons.

Cross-modal challenging: Projection of brain response on stereoscopic image quality ranking

This work presents a novel cross-modality method for acquiring human visual perception on stereoscopic image quality ranking (SIQR), aiming to learn latent biological representations and thinking tokens from brain activity. The core idea is to directly project electroencephalogram (EEG) information onto a SIQR neural network model, rather than simulate the complex neuronal connections. To the end, we establish a multimodal brain-visual dataset, and propose a Cross-GAN model including a binocular attention-based fusion (BAF) image modality encoder, a multiscale spatiotemporal (MST) EEG modality encoder and a multimodal feature generation GAN (MF-GAN). In the BAF image encoder, the progressive interactive attention fusion is designed to highlight the significant regions of the image from monocular view to binocular view, which contributes to the global information description of single view and fused view. In the MST EEG encoder, spectral norm normalization constraint is adopted to assure the Lipschitz continuity, and the multiscale convolution blocks is presented to capture EEG representations from spatiotemporal perspective. Finally, the monomodal image and EEG features are sent to the MF-GAN for creating the optimal brain-visual manifold under the control of the generative loss function. Extensive experiments on the brain-visual multimodality SIQR database prove that the CrossGAN can improve the performance via projecting brain response onto image evaluation model in a cross-modal manner, with no need of EEG trails in application.

A New Generative Model for Textual Descriptions of Medical Images Using Transformers Enhanced with Convolutional Neural Networks.

The automatic generation of descriptions for medical images has sparked increasing interest in the healthcare field due to its potential to assist professionals in the interpretation and analysis of clinical exams. This study explores the development and evaluation of a generalist generative model for medical images. Gaps were identified in the literature, such as the lack of studies that explore the performance of specific models for medical description generation and the need for objective evaluation of the quality of generated descriptions. Additionally, there is a lack of model generalization to different image modalities and medical conditions. To address these issues, a methodological strategy was adopted, combining natural language processing and features extraction from medical images and feeding them into a generative model based on neural networks. The goal was to achieve model generalization across various image modalities and medical conditions. The results showed promising outcomes in the generation of descriptions, with an accuracy of 0.7628 and a BLEU-1 score of 0.5387. However, the quality of the generated descriptions may still be limited, exhibiting semantic errors or lacking relevant details. These limitations could be attributed to the availability and representativeness of the data, as well as the techniques used.

An Underwater Side-Scan Sonar Transfer Recognition Method Based on Crossed Point-to-Point Second-Order Self-Attention Mechanism

Recognizing targets through side-scan sonar (SSS) data by deep learning-based techniques has been particularly challenging. The primary challenge stems from the difficulty and time consumption associated with underwater acoustic data acquisition, which demands systematic explorations to obtain sufficient training samples for accurate deep learning-based models. Moreover, if the sample size of the available data is small, the design of effective target recognition models becomes complex. These challenges have posed significant obstacles to developing accurate SSS-based target recognition methods via deep learning models. However, utilizing multi-modal datasets to enhance the recognition performance of sonar images through knowledge transfer in deep networks appears promising. Owing to the unique statistical properties of various modal images, transitioning between different modalities can significantly increase the complexity of network training. This issue remains unresolved, directly impacting the target transfer recognition performance. To enhance the precision of categorizing underwater sonar images when faced with a limited number of mode types and data samples, this study introduces a crossed point-to-point second-order self-attention (PPCSSA) method based on double-mode sample transfer recognition. In the PPCSSA method, first-order importance features are derived by extracting key horizontal and longitudinal point-to-point features. Based on these features, the self-supervised attention strategy effectively removes redundant features, securing the second-order significant features of SSS images. This strategy introduces a potent low-mode-type small-sample learning method for transfer learning. Classification experiment results indicate that the proposed method excels in extracting key features with minimal training complexity. Moreover, experimental outcomes underscore that the proposed technique enhances recognition stability and accuracy, achieving a remarkable overall accuracy rate of 99.28%. Finally, the proposed method maintains high recognition accuracy even in noisy environments.

State-of-the-Art of Breast Cancer Diagnosis in Medical Images via Convolutional Neural Networks (CNNs).

Early detection of breast cancer is crucial for a better prognosis. Various studies have been conducted where tumor lesions are detected and localized on images. This is a narrative review where the studies reviewed are related to five different image modalities: histopathological, mammogram, magnetic resonance imaging (MRI), ultrasound, and computed tomography (CT) images, making it different from other review studies where fewer image modalities are reviewed. The goal is to have the necessary information, such as pre-processing techniques and CNN-based diagnosis techniques for the five modalities, readily available in one place for future studies. Each modality has pros and cons, such as mammograms might give a high false positive rate for radiographically dense breasts, while ultrasounds with low soft tissue contrast result in early-stage false detection, and MRI provides a three-dimensional volumetric image, but it is expensive and cannot be used as a routine test. Various studies were manually reviewed using particular inclusion and exclusion criteria; as a result, 91 recent studies that classify and detect tumor lesions on breast cancer images from 2017 to 2022 related to the five image modalities were included. For histopathological images, the maximum accuracy achieved was around 99 , and the maximum sensitivity achieved was 97.29 by using DenseNet, ResNet34, and ResNet50 architecture. For mammogram images, the maximum accuracy achieved was 96.52 using a customized CNN architecture. For MRI, the maximum accuracy achieved was 98.33 using customized CNN architecture. For ultrasound, the maximum accuracy achieved was around 99 by using DarkNet-53, ResNet-50, G-CNN, and VGG. For CT, the maximum sensitivity achieved was 96 by using Xception architecture. Histopathological and ultrasound images achieved higher accuracy of around 99 by using ResNet34, ResNet50, DarkNet-53, G-CNN, and VGG compared to other modalities for either of the following reasons: use of pre-trained architectures with pre-processing techniques, use of modified architectures with pre-processing techniques, use of two-stage CNN, and higher number of studies available for Artificial Intelligence (AI)/machine learning (ML) researchers to reference. One of the gaps we found is that only a single image modality is used for CNN-based diagnosis; in the future, a multiple image modality approach can be used to design a CNN architecture with higher accuracy.

Visual attention condenser model for multiple disease detection from heterogeneous medical image modalities

Visual attention condenser model for multiple disease detection from heterogeneous medical image modalities

Novel Approach to Multi-Modal Image Fusion using Modified Convolutional Layers

Multimodal image fusion is an important area of research with various applications in computer vision. This research proposes a modification to convolutional layers by fusing two different modalities of images. A novel architecture that uses adaptive fusion mechanisms to learn the optimal weightage of different modalities at each convolutional layer is introduced in the research. The proposed method is evaluated on a publicly available dataset, and the experimental results show that the performance of the proposed method outperforms state-of-the-art methods in terms of various evaluation metrics.

Multi-modal mining of crowd-sourced data: Efficient provision of humanitarian aid to remote regions affected by natural disasters

Data mining applications have the potential to address current deficiencies in the provision of humanitarian aid in natural disasters. Simultaneous text and image analysis in crowd-sourced data can improve the quality of humanitarian aid information. Specifically, we select Bidirectional Encoder Representations from Transformers (BERT) and its descendant ALBERT as pre-trained deep networks for the text modality, while we choose ConvNeXt, RegNet, and Faster RCNN for the image modality. The developed framework demonstrates its application in classifying humanitarian aid through three key aspects. Firstly, it illustrates the effective performance of ConvNeXt and BERT in the classification of humanitarian aid. Secondly, it investigates the efficiency of generative adversarial networks (GAN) in generating synthetic images for imbalanced input datasets. This approach improves the accuracy, precision, recall, and F1-score of the framework when applied to unseen test data. Finally, the study highlights the potential use of SHapley Additive exPlanations (SHAP) for interpreting the behaviour of the developed framework, supporting the timely classification of humanitarian aid information from crowd-sourced data after natural disasters.

Fully Automated Skull Stripping from Brain Magnetic Resonance Images Using Mask RCNN-Based Deep Learning Neural Networks.

This research comprises experiments with a deep learning framework for fully automating the skull stripping from brain magnetic resonance (MR) images. Conventional techniques for segmentation have progressed to the extent of Convolutional Neural Networks (CNN). We proposed and experimented with a contemporary variant of the deep learning framework based on mask region convolutional neural network (Mask-RCNN) for all anatomical orientations of brain MR images. We trained the system from scratch to build a model for classification, detection, and segmentation. It is validated by images taken from three different datasets: BrainWeb; NAMIC, and a local hospital. We opted for purposive sampling to select 2000 images of T1 modality from data volumes followed by a multi-stage random sampling technique to segregate the dataset into three batches for training (75%), validation (15%), and testing (10%) respectively. We utilized a robust backbone architecture, namely ResNet-101 and Functional Pyramid Network (FPN), to achieve optimal performance with higher accuracy. We subjected the same data to two traditional methods, namely Brain Extraction Tools (BET) and Brain Surface Extraction (BSE), to compare their performance results. Our proposed method had higher mean average precision (mAP) = 93% and content validity index (CVI) = 0.95%, which were better than comparable methods. We contributed by training Mask-RCNN from scratch for generating reusable learning weights known as transfer learning. We contributed to methodological novelty by applying a pragmatic research lens, and used a mixed method triangulation technique to validate results on all anatomical modalities of brain MR images. Our proposed method improved the accuracy and precision of skull stripping by fully automating it and reducing its processing time and operational cost and reliance on technicians. This research study has also provided grounds for extending the work to the scale of explainable artificial intelligence (XAI).

The Dependency of Cochlear Lateral Wall Measurements on Observer and Imaging Type.

Assessment techniques for the cochlear spatial lateral wall are associated with inter-rater variability, but derived clinical recommendations nonetheless offer value for individualized electrode selection. Anatomical variations influence the location of cochlear implant electrodes inside the cochlea. Preoperative planning allows individualization of the electrode based on characterization of the bony lateral wall. The study used publicly available digitized temporal bones based on microslicing and computed tomography. Four experienced observers assessed the lateral wall applying manual tracing, linear regression scaling and elliptic-circular approximation methods in all modalities. Radial and height differences were computed in 90-degree steps from the round window center to the apex. Total length, total angular length, and tonotopic frequencies were computed for each reconstruction. Differences were found most pronounced between assessment methods in vertical direction across observers and imaging modalities. One of the five anatomies was consistently found to be of shorter cochlear duct length with estimation techniques yielding more conservative results compared with manual tracings. Assessment techniques for the bony lateral wall yield method, observer, and image modality related deviations. Automation of the anatomical characterization may offer potential in minimizing inaccuracies. Nonetheless, observers were consistently able to detect a smaller inner ear demonstrating the ability of current methods to contribute to an optimized choice of electrodes based on individual patient anatomy.

Towards face anti-spoofing

ABSTRACT Face anti-spoofing is an important part of the face recognition algorithm that aims to prevent face presentation attacks. To facilitate face anti-spoofing research, this paper introduces a novel method in which a simple model provides enhanced performance on the RGB modality images of the CASIA-SURF dataset. Initially, a simple model based on ResNet-50 architecture is proposed that uses three residual layers of ResNet-50 along with a single neuron at the end for binary classification. Meanwhile, to judge the baseline performance, the proposed Resnet-50-based model is trained on the RGB and Depth images, separately. Later, to increase the performance of the proposed ResNet-50-based model on the RGB modality images, a novel Learning Using Privileged Information (LUPI) algorithm is applied. The LUPI contains Depth images and helps to train the inferior model with the help of a pre-trained superior model. Detailed simulations on the CASIA-SURF dataset reveal that an intelligent combination of the ResNet50 based model with the LUPI paradigm works exceptionally well and significantly improves the performance on the RGB modality of the images. The proposed model is less complex and can be deployed in resource-constrained environments.

BAWGNet: Boundary aware wavelet guided network for the nuclei segmentation in histopathology images

Precise cell nucleus segmentation is very critical in many biologically related analyses and disease diagnoses. However, the variability in nuclei structure, color, and modalities of histopathology images make the automatic computer-aided nuclei segmentation task very difficult. Traditional encoder–decoder based deep learning schemes mainly utilize the spatial domain information that may limit the performance of recognizing small nuclei regions in subsequent downsampling operations. In this paper, a boundary aware wavelet guided network (BAWGNet) is proposed by incorporating a boundary aware unit along with an attention mechanism based on a wavelet domain guidance in each stage of the encoder–decoder output. Here the high-frequency 2 Dimensional discrete wavelet transform (2D-DWT) coefficients are utilized in the attention mechanism to guide the spatial information obtained from the encoder–decoder output stages to leverage the nuclei segmentation task. On the other hand, the boundary aware unit (BAU) captures the nuclei’s boundary information, ensuring accurate prediction of the nuclei pixels in the edge region. Furthermore, the preprocessing steps used in our methodology confirm the data’s uniformity by converting it to similar color statistics. Extensive experimentations conducted on three benchmark histopathology datasets (DSB, MoNuSeg and TNBC) exhibit the outstanding segmentation performance of the proposed method (with dice scores 90.82%, 85.74%, and 78.57%, respectively). Implementation of the proposed architecture is available at https://github.com/tamjidimtiaz/BAWGNet.

An adaptive multi-modal hybrid model for classifying thyroid nodules by combining ultrasound and infrared thermal images

BackgroundTwo types of non-invasive, radiation-free, and inexpensive imaging technologies that are widely employed in medical applications are ultrasound (US) and infrared thermography (IRT). The ultrasound image obtained by ultrasound imaging primarily expresses the size, shape, contour boundary, echo, and other morphological information of the lesion, while the infrared thermal image obtained by infrared thermography imaging primarily describes its thermodynamic function information. Although distinguishing between benign and malignant thyroid nodules requires both morphological and functional information, present deep learning models are only based on US images, making it possible that some malignant nodules with insignificant morphological changes but significant functional changes will go undetected.ResultsGiven the US and IRT images present thyroid nodules through distinct modalities, we proposed an Adaptive multi-modal Hybrid (AmmH) classification model that can leverage the amalgamation of these two image types to achieve superior classification performance. The AmmH approach involves the construction of a hybrid single-modal encoder module for each modal data, which facilitates the extraction of both local and global features by integrating a CNN module and a Transformer module. The extracted features from the two modalities are then weighted adaptively using an adaptive modality-weight generation network and fused using an adaptive cross-modal encoder module. The fused features are subsequently utilized for the classification of thyroid nodules through the use of MLP. On the collected dataset, our AmmH model respectively achieved 97.17% and 97.38% of F1 and F2 scores, which significantly outperformed the single-modal models. The results of four ablation experiments further show the superiority of our proposed method.ConclusionsThe proposed multi-modal model extracts features from various modal images, thereby enhancing the comprehensiveness of thyroid nodules descriptions. The adaptive modality-weight generation network enables adaptive attention to different modalities, facilitating the fusion of features using adaptive weights through the adaptive cross-modal encoder. Consequently, the model has demonstrated promising classification performance, indicating its potential as a non-invasive, radiation-free, and cost-effective screening tool for distinguishing between benign and malignant thyroid nodules. The source code is available at https://github.com/wuliZN2020/AmmH.