Image Modalities Research Articles

Texture Analysis in Volumetric Imaging for Dentomaxillofacial Radiology: Transforming Diagnostic Approaches and Future Directions.

This narrative review explores texture analysis as a valuable technique in dentomaxillofacial diagnosis, providing an advanced method for quantification and characterization of different image modalities. The traditional imaging techniques rely primarily on visual assessment, which may overlook subtle variations in tissue structure. In contrast, texture analysis uses sophisticated algorithms to extract quantitative information from imaging data, thus offering deeper insights into the spatial distribution and relationships of pixel intensities. This process identifies unique "texture signatures", serving as markers for accurately characterizing tissue changes or pathological processes. The synergy between texture analysis and radiomics allows radiologists to transcend traditional size-based or semantic descriptors, offering a comprehensive understanding of imaging data. This method enhances diagnostic accuracy, particularly for the assessment of oral and maxillofacial pathologies. The integration of texture analysis with radiomics expands the potential for precise tissue characterization by moving beyond the limitations of human eye evaluations. This article reviews the current trends and methodologies in texture analysis within the field of dentomaxillofacial imaging, highlights its practical applications, and discusses future directions for research and dental clinical practice.

Spatial attention-based CSR-Unet framework for subdural and epidural hemorrhage segmentation and classification using CT images

BackgroundAutomatic diagnosis and brain hemorrhage segmentation in Computed Tomography (CT) may be helpful in assisting the neurosurgeon in developing treatment plans that improve the patient’s chances of survival. Because medical segmentation of images is important and performing operations manually is challenging, many automated algorithms have been developed for this purpose, primarily focusing on certain image modalities. Whenever a blood vessel bursts, a dangerous medical condition known as intracranial hemorrhage (ICH) occurs. For best results, quick action is required. That being said, identifying subdural (SDH) and epidural haemorrhages (EDH) is a difficult task in this field and calls for a new, more precise detection method.MethodsThis work uses a head CT scan to detect cerebral bleeding and distinguish between two types of dural hemorrhages using deep learning techniques. This paper proposes a rich segmentation approach to segment both SDH and EDH by enhancing segmentation efficiency with a better feature extraction procedure. This method incorporates Spatial attention- based CSR (convolution-SE-residual) Unet, for rich segmentation and precise feature extraction.ResultsAccording to the study’s findings, the CSR based Spatial network performs better than the other models, exhibiting impressive metrics for all assessed parameters with a mean dice coefficient of 0.970 and mean IoU of 0.718, while EDH and SDH dice scores are 0.983 and 0.969 respectively.ConclusionsThe CSR Spatial network experiment results show that it can perform well regarding dice coefficient. Furthermore, Spatial Unet based on CSR may effectively model the complicated in segmentations and rich feature extraction and improve the representation learning compared to alternative deep learning techniques, of illness and medical treatment, to enhance the meticulousness in predicting the fatality.

Multimodal dual perception fusion framework for multimodal affective analysis

The misuse of social platforms and the difficulty in regulating post contents have culminated in a surge of negative sentiments, sarcasms, and the rampant spread of fake news. In response, Multimodal sentiment analysis, sarcasm detection and fake news detection based on image and text have attracted considerable attention recently. Due to that these areas share semantic and sentiment features and confront related fusion challenges in deciphering complex human expressions across different modalities, integrating these multimodal classification tasks that share commonalities across different scenarios into a unified framework is expected to simplify research in sentiment analysis, and enhance the effectiveness of classification tasks involving both semantic and sentiment modeling. Therefore, we consider integral components of a broader spectrum of research known as multimodal affective analysis towards semantics and sentiment, and propose a novel multimodal dual perception fusion framework (MDPF). Specifically, MDPF contains three core procedures: (1) Generating bootstrapping language-image Knowledge to enrich origin modality space, and utilizing cross-modal contrastive learning for aligning text and image modalities to understand underlying semantics and interactions. (2) Designing dynamic connective mechanism to adaptively match image-text pairs and jointly employing gaussian-weighted distribution to intensify semantic sequences. (3) Constructing a cross-modal graph to preserve the structured information of both image and text data and share information between modalities, while introducing sentiment knowledge to refine the edge weights of the graph to capture cross-modal sentiment interaction. We evaluate MDPF on three publicly available datasets across three tasks, and the empirical results demonstrate the superiority of our proposed model.

Cap2Seg: leveraging caption generation for enhanced segmentation of COVID-19 medical images

Incorporating medical text annotations compensates for the quality deficiencies of image data, effectively overcoming the limitations of medical image segmentation. Many existing approaches achieve high-quality segmentation results by integrating text into the image modality. However, these approaches require matched image-text pairs during inference to maintain their performance, and the absence of corresponding text annotations results in degraded model performance. Additionally, these methods often assume that the input text annotations are ideal, overlooking the impact of poor-quality text on model performance in practical scenarios. To address these issues, we propose a novel generative medical image segmentation model, Cap2Seg (Leveraging Caption Generation for Enhanced Segmentation of COVID-19 Medical Images). Cap2Seg not only segments lesion areas but also generates related medical text descriptions, guiding the segmentation process. This design enables the model to perform optimal segmentation without requiring text input during inference. To mitigate the impact of inaccurate text on model performance, we consider the consistency between generated textual features and visual features and introduce the Scale-aware Textual Attention Module (SATaM), which reduces the model’s dependency on irrelevant or misleading text information. Subsequently, we design a word-pixel fusion decoding mechanism that effectively integrates textual features into visual features, ensuring that the text information effectively supplements and enhances the image segmentation task. Extensive experiments on two public datasets, MosMedData+ and QaTa-COV19, demonstrate that our method outperforms the current state-of-the-art models under the same conditions. Additionally, ablation studies have been conducted to demonstrate the effectiveness of each proposed module. The code is available at https://github.com/AllenZzzzzzzz/Cap2Seg.

ESFuse: Weak Edge Structure Perception Network for Infrared and Visible Image Fusion

Infrared and visible image fusion (IVIF) fully integrates the complementary features of different modal images, and the fused image provides a more comprehensive and objective interpretation of the scene compared to each source image, thus attracting extensive attention in the field of computer vision in recent years. However, current fusion methods usually center their attention on the extraction of prominent features, falling short of adequately safeguarding subtle and diminutive structures. To address this problem, we propose an end-to-end unsupervised IVIF method (ESFuse), which effectively enhances fine edges and small structures. In particular, we introduce a two-branch head interpreter to extract features from source images of different modalities. Subsequently, these features are fed into the edge refinement module with the detail injection module (DIM) to obtain the edge detection results of the source image, improving the network’s ability to capture and retain complex details as well as global information. Finally, we implemented a multiscale feature reconstruction module to obtain the final fusion results by combining the output of the DIM with the output of the head interpreter. Extensive IVIF fusion experiments on existing publicly available datasets show that the proposed ESFuse outperforms the state-of-the-art(SOTA) methods in both subjective vision and objective evaluation, and our fusion results perform well in semantic segmentation, target detection, pose estimation and depth estimation tasks. The source code has been availabled.

Identifying Implicit Social Biases in Vision-Language Models

Vision-language models, like CLIP (Contrastive Language Image Pretraining), are becoming increasingly popular for a wide range of multimodal retrieval tasks. However, prior work has shown that large language and deep vision models can learn historical biases contained in their training sets, leading to perpetuation of stereotypes and potential downstream harm. In this work, we conduct a systematic analysis of the social biases that are present in CLIP, with a focus on the interaction between image and text modalities. We first propose a taxonomy of social biases called So-B-It, which contains 374 words categorized across ten types of bias. Each type can lead to societal harm if associated with a particular demographic group. Using this taxonomy, we examine images retrieved by CLIP from a facial image dataset using each word as part of a prompt. We find that CLIP frequently displays undesirable associations between harmful words and specific demographic groups, such as retrieving mostly pictures of Middle Eastern men when asked to retrieve images of a "terrorist". Finally, we conduct an analysis of the source of such biases, by showing that the same harmful stereotypes are also present in a large image-text dataset used to train CLIP models for examples of biases that we find. Our findings highlight the importance of evaluating and addressing bias in vision-language models, and suggest the need for transparency and fairness-aware curation of large pre-training datasets.

Retrospective assessment of HDR brachytherapy dose calculation methods in locally advanced cervical cancer patients: AcurosBV vs. AAPM TG43 formalism.

This retrospective analysis was completed to investigate the use of a model-based dose calculation algorithm (MBDCA) AcurosBV, for use in HDR BT treatments for locally advanced cervical cancer treated with tandem and ovoid applicators with interstitial needles. A cohort of 32 patients receiving post-EBRT HDR brachytherapy boost with a prescription dose of 5.5Gy × 5 fractions to the high-risk clinical target volume (CTVHR) were selected for this study. For standard TG43 dose calculation, applicators were manually digitized on the planning images, while for AcurosBV calculations, solid renderings of Titanium Fletcher Suite Delclos (FSD) applicators included in BrachyVision were matched to those used clinically and Ti needles were manually digitized. The dose was recalculated using Varian's AcurosBV 13.5 and dose-to-medium-in-medium (Dm,m) was reported. EQD2 values for targets and organs at risk were compared between dose calculation formalisms. D90% and D98% values were reported for the high and intermediate-risk CTVs, and values were reported for OARs including bladder, rectum, sigmoid, bowel, and vagina. Due to variability within the patient cohort, the dosimetric impact of AcurosBV was investigated corresponding to planning image modality (CT vs. CBCT), presence of Ti needles, and contrast within vaginal balloons used to stabilize implants. AcurosBV showed lower dosimetric values for all plans compared to TG43. The average±standard deviation of dosimetric reduction in D90% was 4.33±0.09% for CTVHR and 4.12±0.09% for CTVIR. The reduction to OARs was: 4.99±0.15% for bladder, 7.87±0.16% for rectum, 5.79±0.17% for sigmoid, 6.91±0.14% for bowel, and 4.55±0.14% for vagina. AcurosBV should be utilized for HDR BT GYN cases, treated with tandem and ovoid applicators, with high degrees of heterogeneity and calculated in tandem with TG43.

Safety of Magnetic Resonance Imaging in Patients with Cardiac Implantable Electronic Devices.

MRI (magnetic resonance imaging) represents the diagnostic image modality of choice in several conditions. With an increasing number of patients requiring MRI for diagnostic purposes, the issue of safety in patients with cardiac implantable electronic devices (CIED) undergoing this imaging modality will play an ever more important role. The purpose of this study was to assess the safety and device function following MRI in an unrestricted real-world cohort of patients with a wide array of cardiac devices. We conducted a retrospective single-center study including 1010 MRI studies conducted in adult patients (≥18 years) with an implanted CIED treated in the University Hospital of Munich (LMU) between July 2012 and March 2024. Patients with non-MR conditionally labeled leads, abandoned or epicardial leads, as well as lead fragments, were included for analysis. Across a total of 1010 MRIs (920 total MR-conditional device generators) performed in patients with an implanted CIED, there were no deaths, reports of discomfort, palpitations, heating, or ventricular arrythmias in the 24 h following MRI. Only 2/1010 MRIs were followed by a reported atrial arrhythmia within 24 h, both in patients with an MR-conditional pacemaker (PM) device without an abandoned lead. No significant changes in device function following MRI from baseline were observed across all included CIEDs. Lastly, no instances of severe malfunction, such as generator failure, loss of capture, electrical reset, or inappropriate inhibition of pacing, were found in post-MRI interrogation reports across all MRI studies. Based on the analysis of 1010 MRIs undergone by patients with CIEDs, following standardized device interrogation, manufacturer-advised device programming, monitoring of vital function, and manufacturer-advised reprogramming, MRI can be performed safely and without adverse events or changes in device function.

Social Event Classification Based on Multimodal Masked Transformer Network

The key to multimodal social event classification is to fully and accurately utilize the features of both image and text modalities. However, most existing methods have the following limitations: (1) they simply concatenate the image features and text features of the event, and (2) there is irrelevant contextual information between different modalities, which leads to mutual interference. Therefore, it is not enough to only consider the relationship between the modalities of multimodal data, but also the irrelevant contextual information (i.e., regions or words) between the modalities. To overcome these limitations, a novel social event classification method based on multimodal masked transformer network (MMTN) is proposed. A better representation of text and image is learned through an image-text encoding network. Then, the obtained image and text representations are input into the multimodal masked transformer network to fuse the multimodal information, and the relationship between the modalities of multimodal information is modeled by calculating the similarity between the multimodal information, masking the irrelevant context between the modalities. Extensive experiments on two benchmark datasets show that the proposed multimodal masked transformer network model achieves state-of-the-art performance.

A novel Deep High-level Concept-mining Jointing Hashing Model for unsupervised cross-modal retrieval

Unsupervised cross-modal hashing has achieved great success in various information retrieval applications owing to its efficient storage usage and fast retrieval speed. Recent studies have primarily focused on training the hash-encoded networks by calculating a sample-based similarity matrix to improve the retrieval performance. However, there are two issues remain to solve: (1) The current sample-based similarity matrix only considers the similarity between image-text pairs, ignoring the different information densities of each modality, which may introduce additional noise and fail to mine key information for retrieval; (2) Most existing unsupervised cross-modal hashing methods only consider alignment between different modalities, while ignoring consistency between each modality, resulting in semantic conflicts. To tackle these challenges, a novel Deep High-level Concept-mining Jointing Hashing (DHCJH) model for unsupervised cross-modal retrieval is proposed in this study. DHCJH is able to capture the essential high-level semantic information from image modalities and integrate into the text modalities to improve the accuracy of guidance information. Additionally, a new hashing loss with a regularization term is introduced to avoid the cross-modal semantic collision and false positive pairs problems. To validate the proposed method, extensive comparison experiments on benchmark datasets are conducted. Experimental findings reveal that DHCJH achieves superior performance in both accuracy and efficiency. The code of DHCJH is available at Github.

BI‐LAVA: Biocuration With Hierarchical Image Labelling Through Active Learning and Visual Analytics

AbstractIn the biomedical domain, taxonomies organize the acquisition modalities of scientific images in hierarchical structures. Such taxonomies leverage large sets of correct image labels and provide essential information about the importance of a scientific publication, which could then be used in biocuration tasks. However, the hierarchical nature of the labels, the overhead of processing images, the absence or incompleteness of labelled data and the expertise required to label this type of data impede the creation of useful datasets for biocuration. From a multi‐year collaboration with biocurators and text‐mining researchers, we derive an iterative visual analytics and active learning (AL) strategy to address these challenges. We implement this strategy in a system called BI‐LAVA—Biocuration with Hierarchical Image Labelling through Active Learning and Visual Analytics. BI‐LAVA leverages a small set of image labels, a hierarchical set of image classifiers and AL to help model builders deal with incomplete ground‐truth labels, target a hierarchical taxonomy of image modalities and classify a large pool of unlabelled images. BI‐LAVA's front end uses custom encodings to represent data distributions, taxonomies, image projections and neighbourhoods of image thumbnails, which help model builders explore an unfamiliar image dataset and taxonomy and correct and generate labels. An evaluation with machine learning practitioners shows that our mixed human–machine approach successfully supports domain experts in understanding the characteristics of classes within the taxonomy, as well as validating and improving data quality in labelled and unlabelled collections.

Building a Multimodal Dataset of Academic Paper for Keyword Extraction

ABSTRACTUp to this point, keyword extraction task typically relies solely on textual data. Neglecting visual details and audio features from image and audio modalities leads to deficiencies in information richness and overlooks potential correlations, thereby constraining the model's ability to learn representations of the data and the accuracy of model predictions. Furthermore, the currently available multimodal datasets for keyword extraction task are particularly scarce, further hindering the progress of research on multimodal keyword extraction task. Therefore, this study constructs a multimodal dataset of academic paper consisting of 1,000 samples, with each sample containing paper text, images, audios and keywords. Based on unsupervised and supervised methods of keyword extraction, experiments are conducted using textual data from papers, as well as text extracted from images and audio. The aim is to investigate the differences in performance in keyword extraction task with respect to different modal information and the fusion of multimodal information. The experimental results indicate that text from different modalities exhibits distinct characteristics in the model. The concatenation of paper text, image text and audio text can effectively enhance the keyword extraction performance of academic papers.

Enhancing ultrasonic attenuation images through multi-frequency coupling with total nuclear variation.

Quantitative ultrasound is a non-invasive image modality that numerically characterizes tissues for medical diagnosis using acoustical parameters, such as the attenuation coefficient slope. A previous study introduced the total variation spectral log difference (TVSLD) method, which denoises spectral log ratios on a single-channel basis without inter-channel coupling. Therefore, this work proposes a multi-frequency joint framework by coupling information across frequency channels exploiting structural similarities among the spectral ratios to increase the quality of the attenuation images. A modification based on the total nuclear variation (TNV) was considered. Metrics were compared to the TVSLD method with simulated and experimental phantoms and two samples of fibroadenoma in vivo breast tissue. The TNV demonstrated superior performance, yielding enhanced attenuation coefficient slope maps with fewer artifacts at boundaries and a stable error. In terms of the contrast-to-noise ratio enhancement, the TNV approach obtained an average percentage improvement of 34% in simulation, 38% in the experimental phantom, and 89% in two in vivo breast tissue samples compared to TVSLD, showing potential to enhance visual clarity and depiction of attenuation images.

Dual phase hierarchical brain tumour detection and segmentation using unet based skip guidance residual convolutional transformer

Brain tumour segmentation and classification is an important approach to understanding tumour characteristics effectively. Accurate lesion segmentation requires many image modalities with varied contrasts. As a result, manual segmentation may not be suitable for large-scale research. The research has shown interest in developing various learning algorithms to detect and segment brain tumour (BT) images accurately based on the set of experimental images. Some of the issues noticed in the existing models, such as high computational complexity, high processing time, and less accuracy, were identified. In order to overcome these existing issues, a novel Dual-phase channel attentional Residual convolutional transformer Unet model (DCARCTUnet) is introduced for segmenting brain tumours. Initially, the input images are passed into the pre-processing stage to remove noise and blur. The Efficient Net model identified the pre-processed image as either normal or tumour. The proposed segmentation model can be integrated into three stages, namely dual phase channel attention module (DCAM), residual convolutional transformer Unet model and Pyramid skip guidance module (PSM). The proposed method uses the BRATS 2020–2021 dataset to analyze the results accurately. The results showed that the proposed model has a dice coefficient value of 0.99 and a Jaccard coefficient value of 0.98. Furthermore, performance analysis is carried out to demonstrate the effectiveness of the proposed model.

DELR-Net: a network for 3D multimodal medical image registration in more lightweight application scenarios.

3D multimodal medical image deformable registration plays a significant role in medical image analysis and diagnosis. However, due to the substantial differences between images of different modalities, registration is challenging and requires high computational costs. Deep learning-based registration methods face these challenges. The primary aim of this paper is to design a 3D multimodal registration network that ensures high-quality registration results while reducing the number of parameters. This study designed a Dual-Encoder More Lightweight Registration Network (DELR-Net). DELR-Net is a low-complexity network that integrates Mamba and ConvNet. The State Space Sequence Module and the Dynamic Large Kernel block are used as the main components of the dual encoders, while the Dynamic Feature Fusion block is used as the main component of the decoder. This study conducted experiments on 3D brain MR images and abdominal MR and CT images. Compared to existing registration methods, DELR-Net achieved better registration results while maintaining a lower number of parameters. Additionally, generalization experiments on other modalities showed that DELR-Net has superior generalization capabilities. DELR-Net significantly improves the limitations of 3D multimodal medical image deformable registration, achieving better registration performance with fewer parameters.

GEXMERT: Geometrically enhanced cross-modality encoder representations from transformers inspired by higher-order visual percepts

Owing to the rapid progress made in transformer-based models and multi-modal fusion strategies, the performance of multi-modal networks has remarkably improved in recent years. However, intrinsic connections among individual modalities are crucial in multi-modal fusion strategies, presenting substantial opportunities for further enhancement of multi-modal perception in this field. Taking inspiration from the biological mechanisms of the human multi-modal cognitive system, we proposed a novel cross-modality fusion network. This network leverages the lateral occipital complex, which is a region responsible for object-shape perception in human vision and integrates an auxiliary modality derived using the conventional multi-stream model. This integration for learning complementary information across image and language modalities effectively captures the biological mechanisms of human processing of higher-order visual percepts through parameter updates of the intermediate modality to optimize the suitability for further fusion. To evaluate the effectiveness of our approach, we conducted experiments under various settings. Our statistical analyses demonstrate a considerable enhancement in the performance of vision-language tasks, thereby validating biological plausibility of our model.

Prompt-Based Modality Bridging for Unified Text-to-Face Generation and Manipulation

Text-driven face image generation and manipulation are significant tasks. However, such tasks are quite challenging due to the gap between text and image modalities. It is difficult to utilize current methods to deal with both of the two problems because these methods are usually designed for one certain task, limiting their application in real scenarios. To address the two problems in one framework, we propose a U nified P rompt-based C ross- M odal Frame work (UPCM-Frame) to bridge the gap between the text modality and image modality with CLIP and StyleGAN, which are two large-scale pre-trained models. The proposed framework is combined with two main modules: a Text Embedding-to-Image Embedding projection module based on a special prompt embedding pair, and a projection module mapping Image Embeddings to semantically aligned StyleGAN Embeddings which can be used in both image generation and manipulation. The proposed framework is able to handle complicated descriptions and generate impressive results with high quality due to the utilization of large-scale pre-trained models. In order to demonstrate the effectiveness of the proposed method in the two tasks, we conduct experiments to evaluate the results of our method both quantitatively and qualitatively.

Few-Shot Image Classification of Crop Diseases Based on Vision-Language Models.

Accurate crop disease classification is crucial for ensuring food security and enhancing agricultural productivity. However, the existing crop disease classification algorithms primarily focus on a single image modality and typically require a large number of samples. Our research counters these issues by using pre-trained Vision-Language Models (VLMs), which enhance the multimodal synergy for better crop disease classification than the traditional unimodal approaches. Firstly, we apply the multimodal model Qwen-VL to generate meticulous textual descriptions for representative disease images selected through clustering from the training set, which will serve as prompt text for generating classifier weights. Compared to solely using the language model for prompt text generation, this approach better captures and conveys fine-grained and image-specific information, thereby enhancing the prompt quality. Secondly, we integrate cross-attention and SE (Squeeze-and-Excitation) Attention into the training-free mode VLCD(Vision-Language model for Crop Disease classification) and the training-required mode VLCD-T (VLCD-Training), respectively, for prompt text processing, enhancing the classifier weights by emphasizing the key text features. The experimental outcomes conclusively prove our method's heightened classification effectiveness in few-shot crop disease scenarios, tackling the data limitations and intricate disease recognition issues. It offers a pragmatic tool for agricultural pathology and reinforces the smart farming surveillance infrastructure.

A cross-model hierarchical interactive fusion network for end-to-end multimodal aspect-based sentiment analysis

For the aspect-based sentiment analysis task, traditional works are only for text modality. However, in social media scenarios, texts often contain abbreviations, clerical errors, or grammatical errors, which invalidate traditional methods. In this study, the cross-model hierarchical interactive fusion network incorporating an end-to-end approach is proposed to address this challenge. In the network, a feature attention module and a feature fusion module are proposed to obtain the multimodal interaction feature between the image modality and the text modality. Through the attention mechanism and gated fusion mechanism, these two modules realize the auxiliary function of image in the text-based aspect-based sentiment analysis task. Meanwhile, a boundary auxiliary module is used to explore the dependencies between two core subtasks of the aspect-based sentiment analysis. Experimental results on two publicly available multi-modal aspect-based sentiment datasets validate the effectiveness of the proposed approach.

MEEAFusion: Multi-Scale Edge Enhancement and Joint Attention Mechanism Based Infrared and Visible Image Fusion.

Infrared and visible image fusion can integrate rich edge details and salient infrared targets, resulting in high-quality images suitable for advanced tasks. However, most available algorithms struggle to fully extract detailed features and overlook the interaction of complementary features across different modal images during the feature fusion process. To address this gap, this study presents a novel fusion method based on multi-scale edge enhancement and a joint attention mechanism (MEEAFusion). Initially, convolution kernels of varying scales were utilized to obtain shallow features with multiple receptive fields unique to the source image. Subsequently, a multi-scale gradient residual block (MGRB) was developed to capture the high-level semantic information and low-level edge texture information of the image, enhancing the representation of fine-grained features. Then, the complementary feature between infrared and visible images was defined, and a cross-transfer attention fusion block (CAFB) was devised with joint spatial attention and channel attention to refine the critical supplemental information. This allowed the network to obtain fused features that were rich in both common and complementary information, thus realizing feature interaction and pre-fusion. Lastly, the features were reconstructed to obtain the fused image. Extensive experiments on three benchmark datasets demonstrated that the MEEAFusion proposed in this research has considerable strengths in terms of rich texture details, significant infrared targets, and distinct edge contours, and it achieves superior fusion performance.