Multimodal Model Research Articles

Study on the classification of benign and malignant breast lesions using a multi-sequence breast MRI fusion radiomics and deep learning model

PurposeTo develop a multi-modal model combining multi-sequence breast MRI fusion radiomics and deep learning for the classification of benign and malignant breast lesions, to assist clinicians in better selecting treatment plans. MethodsA total of 314 patients who underwent breast MRI examinations were included. They were randomly divided into training, validation, and test sets in a ratio of 7:1:2. Subsequently, features of T1-weighted images (T1WI), T2-weighted images (T2WI), and dynamic contrast-enhanced MRI (DCE-MRI) were extracted using the convolutional neural network ResNet50 for fusion, and then combined with radiomic features from the three sequences. The following models were established: T1 model, T2 model, DCE model, DCE_T1_T2 model, and DCE_T1_T2_rad model. The performance of the models was evaluated by the area under the receiver operating characteristic (ROC) curve (AUC), accuracy, sensitivity, specificity, positive predictive value, and negative predictive value. The differences between the DCE_T1_T2_rad model and the other four models were compared using the Delong test, with a P-value < 0.05 considered statistically significant. ResultsThe five models established in this study performed well, with AUC values of 0.53 for the T1 model, 0.62 for the T2 model, 0.79 for the DCE model, 0.94 for the DCE_T1_T2 model, and 0.98 for the DCE_T1_T2_rad model. The DCE_T1_T2_rad model showed statistically significant differences (P < 0.05) compared to the other four models. ConclusionThe use of a multi-modal model combining multi-sequence breast MRI fusion radiomics and deep learning can effectively improve the diagnostic performance of breast lesion classification.

Are multimodal large language models a viable option for reliable, low-cost football analysis?

As the reasoning, vision and data analysis capabilities of multimodal large language models continue to develop rapidly, with competing models outcompeting their predecessors on many benchmarks, the question of how these capabilities can be used arises. One potential use case lies in sports analytics: a rapidly developing field which continues to rely more heavily on advanced technical methods than in previous years. This paper explores how state-of-the-art large language models can be utilised in football player analysis by determining the reliability of information provided in three crucial areas in the sports analysis field: data collection, data analysis and data visualisation to help determine the feasibility of using such models to provide reliable player analysis at lower costs than current methods.

Impact of metadata in multimodal classification of bone tumours

The accurate classification of bone tumours is crucial for guiding clinical decisions regarding treatment and follow-up. However, differentiating between various tumour types is challenging due to the rarity of certain entities, high intra-class variability, and limited training data in clinical practice. This study proposes a multimodal deep learning model that integrates clinical metadata and X-ray imaging to improve the classification of primary bone tumours. The dataset comprises 1,785 radiographs from 804 patients collected between 2000 and 2020, including metadata such as age, affected bone site, tumour position, and gender. Ten tumour types were selected, with histopathology or tumour board decisions serving as the reference standard.MethodsOur model is based on the NesT image classification model and a multilayer perceptron with a joint fusion architecture. Descriptive statistics included incidence and percentage ratios for discrete parameters, and mean, standard deviation, median, and interquartile range for continuous parameters.ResultsThe mean age of the patients was 33.62 ± 18.60 years, with 54.73% being male. Our multimodal deep learning model achieved 69.7% accuracy in classifying primary bone tumours, outperforming the Vision Transformer model by five percentage points. SHAP values indicated that age had the most substantial influence among the considered metadata.ConclusionThe joint fusion approach developed in this study, integrating clinical metadata and imaging data, outperformed state-of-the-art models in classifying primary bone tumours. The use of SHAP values provided insights into the impact of different metadata on the model’s performance, highlighting the significant role of age. This approach has potential implications for improving diagnostic accuracy and understanding the influence of clinical factors in tumour classification.

Peak Identification in Evolutionary Multimodal Optimization: Model, Algorithms, and Metrics.

In this paper, we present a two-phase multimodal optimization model designed to efficiently and accurately identify multiple optima. The first phase employs a population-based search algorithm to locate potential optima, while the second phase introduces a novel peak identification (PI) procedure to filter out non-optimal solutions, ensuring that each identified solution represents a distinct optimum. This approach not only enhances the effectiveness of multimodal optimization but also addresses the issue of redundant solutions prevalent in existing algorithms. We propose two PI algorithms: HVPI, which uses a hill-valley approach to distinguish between optima, without requiring prior knowledge of niche radii; and HVPIC, which integrates HVPI with bisecting K-means clustering to reduce the number of fitness evaluations (FEs). The performance of these algorithms was evaluated using the F-measure, a comprehensive metric that accounts for both the accuracy and redundancy in the solution set. Extensive experiments on a suite of benchmark functions and engineering problems demonstrated that our proposed algorithms achieved a high precision and recall, significantly outperforming traditional methods.

Two-Stage Multimodal Method for Predicting Intramuscular Fat in Pigs

Intramuscular fat (IMF) content significantly influences pork tenderness, flavor, and juiciness. Maintaining an optimal IMF range not only enhances nutritional value but also improves the taste of pork products. However, traditional IMF measurement methods are often invasive and time-consuming. Ultrasound imaging technology offers a non-destructive solution capable of predicting IMF content and assessing backfat thickness as well as longissimus dorsi muscle area size. A two-stage multimodal network model was developed in this study. First, using B-mode ultrasound images, we employed the UNetPlus segmentation network to accurately delineate the longissimus dorsi muscle area. Subsequently, we integrated data on backfat thickness and longissimus dorsi muscle area to create a multimodal input for IMF content prediction using our model. The results indicate that UNetPlus achieves a 94.17% mean Intersection over Union (mIoU) for precise longissimus dorsi muscle area segmentation. The multimodal network achieves an R2 of 0.9503 for IMF content prediction, with Spearman and Pearson correlation coefficients of 0.9683 and 0.9756, respectively, all within a compact model size of 4.96 MB. This study underscores the efficacy of combining segmented longissimus dorsi muscle images with data on backfat thickness and muscle area in a two-stage multimodal approach for predicting IMF content.

Substantive and Emotional: The Anthropomorphic Evolution of Generative AITaking GPT-4o as an Example

Under the background of the rapid development of artificial intelligence technology, "personal publishing" represented by blog in the past has transformed into "human-computer co-creation", showing a new human-computer interaction relationship, and generative artificial intelligence has become the "collaborator" of human content creation. The flagship multimodal large language model "anthropomorphism" represented by GPT-4o is evolving from "linguistic anthropomorphism" to "emotional anthropomorphism". Based on the anthropomorphic theory, this paper aims to reveal that GPT-4o, while demonstrating its linguistic intelligence in the human-machine co-created dialogue, simulates human emotional response, showing empathy and "resonance", that is, the "anthropomorphic" process at the two levels of "words have things" and "words have feelings". It also reflects on the potential ethical challenges of the anthropomorphic evolution of generative AI to future human society.

Revolution or risk?-Assessing the potential and challenges of GPT-4V in radiologic image interpretation.

ChatGPT-4 Vision (GPT-4V) is a state-of-the-art multimodal large language model (LLM) that may be queried using images. We aimed to evaluate the tool's diagnostic performance when autonomously assessing clinical imaging studies. A total of 206 imaging studies (i.e., radiography (n = 60), CT (n = 60), MRI (n = 60), and angiography (n = 26)) with unequivocal findings and established reference diagnoses from the radiologic practice of a large university hospital were accessed. Readings were performed uncontextualized, with only the image provided, and contextualized, with additional clinical and demographic information. Responses were assessed along multiple diagnostic dimensions and analyzed using appropriate statistical tests. With its pronounced propensity to favor context over image information, the tool's diagnostic accuracy improved from 8.3% (uncontextualized) to 29.1% (contextualized, first diagnosis correct) and 63.6% (contextualized, correct diagnosis among differential diagnoses) (p ≤ 0.001, Cochran's Q test). Diagnostic accuracy declined by up to 30% when 20 images were re-read after 30 and 90 days and seemed unrelated to the tool's self-reported confidence (Spearman's ρ = 0.117 (p = 0.776)). While the described imaging findings matched the suggested diagnoses in 92.7%, indicating valid diagnostic reasoning, the tool fabricated 258 imaging findings in 412 responses and misidentified imaging modalities or anatomic regions in 65 images. GPT-4V, in its current form, cannot reliably interpret radiologic images. Its tendency to disregard the image, fabricate findings, and misidentify details, especially without clinical context, may misguide healthcare providers and put patients at risk. Question Can Generative Pre-trained Transformer 4 Vision (GPT-4V) interpret radiologic images-with and without clinical context? Findings GPT-4V performed poorly, demonstrating diagnostic accuracy rates of 8% (uncontextualized), 29% (contextualized, most likely diagnosis correct), and 64% (contextualized, correct diagnosis among differential diagnoses). Clinical relevance The utility of commercial multimodal large language models, such as GPT-4V, in radiologic practice is limited. Without clinical context, diagnostic errors and fabricated findings may compromise patient safety and misguide clinical decision-making. These models must be further refined to be beneficial.

Multimodal deep learning radiomics model for predicting postoperative progression in solid stage I non-small cell lung cancer

PurposeTo explore the application value of a multimodal deep learning radiomics (MDLR) model in predicting the risk status of postoperative progression in solid stage I non-small cell lung cancer (NSCLC).Materials and MethodsA total of 459 patients with histologically confirmed solid stage I NSCLC who underwent surgical resection in our institution from January 2014 to September 2019 were reviewed retrospectively. At another medical center, 104 patients were reviewed as an external validation cohort according to the same criteria. A univariate analysis was conducted on the clinicopathological characteristics and subjective CT findings of the progression and non-progression groups. The clinicopathological characteristics and subjective CT findings that exhibited significant differences were used as input variables for the extreme learning machine (ELM) classifier to construct the clinical model. We used the transfer learning strategy to train the ResNet18 model, used the model to extract deep learning features from all CT images, and then used the ELM classifier to classify the deep learning features to obtain the deep learning signature (DLS). A MDLR model incorporating clinicopathological characteristics, subjective CT findings and DLS was constructed. The diagnostic efficiencies of the clinical model, DLS model and MDLR model were evaluated by the area under the curve (AUC).ResultsUnivariate analysis indicated that size (p = 0.004), neuron-specific enolase (NSE) (p = 0.03), carbohydrate antigen 19 − 9 (CA199) (p = 0.003), and pathological stage (p = 0.027) were significantly associated with the progression of solid stage I NSCLC after surgery. Therefore, these clinical characteristics were incorporated into the clinical model to predict the risk of progression in postoperative solid-stage NSCLC patients. A total of 294 deep learning features with nonzero coefficients were selected. The DLS in the progressive group was (0.721 ± 0.371), which was higher than that in the nonprogressive group (0.113 ± 0.350) (p < 0.001). The combination of size、NSE、CA199、pathological stage and DLS demonstrated the superior performance in differentiating postoperative progression status. The AUC of the MDLR model was 0.885 (95% confidence interval [CI]: 0.842–0.927), higher than that of the clinical model (0.675 (95% CI: 0.599–0.752)) and DLS model (0.882 (95% CI: 0.835–0.929)). The DeLong test and decision in curve analysis revealed that the MDLR model was the most predictive and clinically useful model.ConclusionMDLR model is effective in predicting the risk of postoperative progression of solid stage I NSCLC, and it is helpful for the treatment and follow-up of solid stage I NSCLC patients.

Spectral broadening and vibronic dynamics of the S2 state of canthaxanthin in the orange carotenoid protein

We have performed a series of broadband multidimensional electronic spectroscopy experiments to probe the electronic and vibrational dynamics of the canthaxanthin chromophore of the Orange Carotenoid Protein (OCP) from Synechocystis sp. PCC 6803 in its photoactivated red state, OCPR. Cross-peaks observed below the diagonal of the two-dimensional electronic spectrum indicate that absorption transitions prepare the bright S2 state of the ketocarotenoid canthaxanthin near to a sequence of conical intersections, allowing passage to the dark S1 state via the Sx intermediate in &amp;lt;50 fs. Rapid damping of excited-state coherent wavepacket motions suggests that the branching coordinates of the conical intersections include out-of-plane deformation and C=C stretching coordinates of the π-conjugated isoprenoid backbone. The unusual proximity of the Franck–Condon S2 state structure to the conical intersections with Sx and S1 suggests that the protein surroundings of canthaxanthin prepare it to function as an excitation energy trap in the OCPR–phycobilisome complex. Numerical simulations using the multimode Brownian oscillator model demonstrate that the ground-state absorption spectrum of OCPR overlaps with the fluorescence emission spectrum of allophycocyanin due to spectral broadening derived especially from the intramolecular motions of the canthaxanthin chromophore in its binding site.

Multimodal Fusion of EEG and Audio Spectrogram for Major Depressive Disorder Recognition Using Modified DenseNet121.

This study investigates the classification of Major Depressive Disorder (MDD) using electroencephalography (EEG) Short-Time Fourier-Transform (STFT) spectrograms and audio Mel-spectrogram data of 52 subjects. The objective is to develop a multimodal classification model that integrates audio and EEG data to accurately identify depressive tendencies. We utilized the Multimodal open dataset for Mental Disorder Analysis (MODMA) and trained a pre-trained Densenet121 model using transfer learning. Features from both the EEG and audio modalities were extracted and concatenated before being passed through the final classification layer. Additionally, an ablation study was conducted on both datasets separately. The proposed multimodal classification model demonstrated superior performance compared to existing methods, achieving an Accuracy of 97.53%, Precision of 98.20%, F1 Score of 97.76%, and Recall of 97.32%. A confusion matrix was also used to evaluate the model's effectiveness. The paper presents a robust multimodal classification approach that outperforms state-of-the-art methods with potential application in clinical diagnostics for depression assessment.

L’émergence d’un nouveau modèle de traduction – théorie et pratique de la traduction multimodale sur l’exemple de la traduction en français de l’album Kłopot d’Iwona Chmielewska

Translation activity is traditionally perceived as monomodal, focused on the written or spoken word. However, the evolution of contemporary societies, increasingly visual, often pushes the translator to work on a multimodal product, where language is only one of the modes of communication, hence the emergence of models of multimodal translation. The present contribution studies the literary multimodal translation model based on the album Kłopot by Iwona Chmielewska, world-renowned author and illustrator. The contribution hopes to demonstrate the challenges and opportunities of the multimodal translation workshop, seen as a growing professional activity in today’s multimodal societies.

Machine Learning-Enabled Nanoscale Phase Prediction in Engineered Poly(Vinylidene Fluoride).

Engineered poly(vinylidene fluoride) (PVDF) with its diverse crystalline phases plays a crucial role in determining the performance of devices in piezo-, pyro-, ferro- and tribo-electric applications, indicating the importance of distinct phase-detection in defining the structure-property relation. However, traditional characterization techniques struggle to effectively distinguish these phases, thereby failing to offer complete information. In this study, multimodal data-driven techniques have been employed for distinguishing different phases with a machine learning (ML) approach. This developed multimode model has been trained from empirical to theoretical data and demonstrates a classification accuracy of >94%, 15% more noise resilience, and 11% more accuracy from unimodality. Thus, from conception to validation, an alternative approach is provided to autonomously distinguish the different PVDF phases and eschew repetitive experiments that saved resources, thus accelerating the process of materials selection in various applications.

Evolution and future directions of Artificial Intelligence Generated Content (AIGC): A comprehensive review

Artificial Intelligence Generated Content (AIGC) has rapidly evolved, revolutionizing the creation of text, images, audio, and video content. Despite these advancements, research on the development process of AIGC technology remains scarce, necessitating a systematic discussion of its current state and future directions. So this paper delves into the significant advancements and foundational technologies driving AIGC, emphasizing the contributions of state-of-the-art models such as DALL-E 3 [1] and Sora [2]. We analyze the evolution of generative models from single-modal approaches to the current multimodal generative models. The paper further explores the application prospects of AIGC across various domains such as office work, art, education, and film, while addressing the existing limitations and challenges in the field. We propose potential improvement directions, including more efficient model architectures and enhanced multimodal capabilities. Emphasis is placed on the environmental impact of AIGC technologies and the need for sustainable practices. Our comprehensive review aims to provide researchers and professionals with a deeper understanding of AIGC, inspiring further exploration and innovation in this transformative domain.

A comprehensive investigation of multimodal deep learning fusion strategies for breast cancer classification

In breast cancer research, diverse data types and formats, such as radiological images, clinical records, histological data, and expression analysis, are employed. Given the intricate nature of natural phenomena, relying on the features of a single modality is seldom sufficient for comprehensive analysis. Therefore, it is possible to guarantee medical relevance and achieve improved clinical outcomes by combining several modalities. The presen study carefully maps and reviews 47 primary articles from six well-known digital libraries that were published between 2018 and 2023 for breast cancer classification based on multimodal deep learning fusion (MDLF) techniques. This systematic literature review encompasses various aspects, including the medical modalities combined, the datasets utilized in these studies, the techniques, models, and architectures used in MDLF and it also discusses the advantages and limitations of each approach. The analysis of selected papers has revealed a compelling trend: the emergence of new modalities and combinations that were previously unexplored in the context of breast cancer classification. This exploration has not only expanded the scope of predictive models but also introduced fresh perspectives for addressing diverse targets, ranging from screening to diagnosis and prognosis. The practical advantages of MDLF are evident in its ability to enhance the predictive capabilities of machine learning models, resulting in improved accuracy across diverse applications. The prevalence of deep learning models underscores their success in autonomously discerning complex patterns, offering a substantial departure from traditional machine learning approaches. Furthermore, the paper explores the challenges and future directions in this field, including the need for larger datasets, the use of ensemble learning methods, and the interpretation of multimodal models.

Early prediction of sudden cardiac death using multimodal fusion of ECG Features extracted from Hilbert–Huang and wavelet transforms with explainable vision transformer and CNN models

Background and Objective:Sudden cardiac death (SCD) is a critical health issue characterized by the sudden failure of heart function, often caused by ventricular fibrillation (VF). Early prediction of SCD is crucial to enable timely interventions. However, current methods predict SCD only a few minutes before its onset, limiting intervention time. This study aims to develop a deep learning-based model for the early prediction of SCD using electrocardiography (ECG) signals. Methods:A multimodal explainable deep learning-based model is developed to analyze ECG signals at discrete intervals ranging from 5 to 30 min before SCD onset. The raw ECG signals, 2D scalograms generated through wavelet transform and 2D Hilbert spectrum generated through Hilbert–Huang transform (HHT) of ECG signals were applied to multiple deep learning algorithms. For raw ECG, a combination of 1D-convolutional neural networks (1D-CNN) and long short-term memory networks were employed for feature extraction and temporal pattern recognition. Besides, to extract and analyze features from scalograms and Hilbert spectra, Vision Transformer (ViT) and 2D-CNN have been used. Results:The developed model achieved high performance, with accuracy, precision, recall and F1-score of 98.81%, 98.83%, 98.81%, and 98.81% respectively to predict SCD onset 30 min in advance. Further, the proposed model can accurately classify SCD patients and normal controls with 100% accuracy. Thus, the proposed method outperforms the existing state-of-the-art methods. Conclusions:The developed model is capable of capturing diverse patterns on ECG signals recorded at multiple discrete time intervals (at 5-minute increments from 5 min to 30 min) prior to SCD onset that could discriminate for SCD. The proposed model significantly improves early SCD prediction, providing a valuable tool for continuous ECG monitoring in high-risk patients.

Multimodal apparent diffusion MRI model in noninvasive evaluation of breast cancer and Ki-67 expression

BackgroundTo assess the capability of multimodal apparent diffusion (MAD) weighted magnetic resonance imaging (MRI) to distinguish between malignant and benign breast lesions, and to predict Ki-67 expression level in breast cancer.MethodsThis retrospective study was conducted with 93 patients who had postoperative pathology-confirmed breast cancer or benign breast lesions. MAD images were acquired using a 3.0 T MRI scanner with 16 b values. The MAD parameters, as flow (fF, DF), unimpeded (fluid) (fUI), hindered (fH, DH, and αH), and restricted (fR, DR), were calculated. The differences of the parameters were compared by Mann–Whitney U test between the benign/malignant lesions and high/low Ki-67 expression level. The diagnostic performance was assessed by the area under the receiver operating characteristic curve (AUC).ResultsThe fR in the malignant lesions was significantly higher than in the benign lesions (P = 0.001), whereas the fUI and DH were found to be significantly lower (P = 0.007 and P < 0.001, respectively). Compared with individual parameter in differentiating malignant from benign breast lesions, the combination parameters of MAD (fR, DH, and fUI) provided the highest AUC (0.851). Of the 73 malignant lesions, 42 (57.5%) were assessed as Ki-67 low expression and 31 (42.5%) were Ki-67 high expression. The Ki-67 high status showed lower DH, higher DF and higher αH (P < 0.05). The combination parameters of DH, DF, and αH provided the highest AUC (0.691) for evaluating Ki-67 expression level.ConclusionsMAD weighted MRI is a useful method for the breast lesions diagnostics and the preoperative prediction of Ki-67 expression level.

Emerging challenges in the application of artificial intelligence for the eye disease screening in Chinese primary healthcare institutions

Breakthroughs have been achieved recently in the application of artificial intelligence (AI) for the eye disease screening in Chinese primary healthcare institutions, but challenges have also emerged. First, AI software has continuously evolved, expanding the range of eye diseases that can be screened, enhancing diagnostic accuracy, and progressing towards predicting the course of eye diseases. However, inadequate infrastructure such as 5G and a shortage of specialized personnel have hindered the coverage of screenings. Second, while the cost-effectiveness of AI is well-established, new screening models have impacted the equity of screenings. It is essential to tailor AI application models to the specific context of China. Third, AI screening guidelines have been increasingly improved, providing direction for AI development and reference for the promotion and application of AI technologies. Nonetheless, high-quality empirical research is urgently needed to provide scientific evidence for policymaking related to AI in the eye disease screening. Therefore, it is suggested to develop multimodal AI models that integrate basic data such as symptoms and medical history with simple ophthalmic examinations, to accelerate the construction of infrastructure like 5G and focus on cultivating interdisciplinary talents, to explore suitable service systems and models for the large-scale eye disease screening tailored to local conditions, and to conduct long-term, multi-center, empirical studies.

Leveraging Multimodal Large Language Models (MLLMs) for Enhanced Object Detection and Scene Understanding in Thermal Images for Autonomous Driving Systems

Leveraging Multimodal Large Language Models (MLLMs) for Enhanced Object Detection and Scene Understanding in Thermal Images for Autonomous Driving Systems

Comparison of deep-learning multimodality data fusion strategies in mandibular osteoradionecrosis NTCP modelling using clinical variables and radiation dose distribution volumes

Objective.Normal tissue complication probability (NTCP) modelling is rapidly embracing deep learning (DL) methods, acknowledging the importance of spatial dose information. Finding effective ways to combine information from radiation dose distribution maps (dosiomics) and clinical data involves technical challenges and requires domain knowledge. We propose different multi-modality data fusion strategies to facilitate future DL-based NTCP studies.Approach.Early, joint and late DL multi-modality fusion strategies were compared using clinical and mandibular radiation dose distribution volumes. These were contrasted with single-modality models: a random forest trained on non-image data (clinical, demographic and dose-volume metrics) and a 3D DenseNet-40 trained on image data (mandibular dose distribution maps). The study involved a matched cohort of 92 osteoradionecrosis cases and 92 controls from a single institution.Main results.The late fusion model exhibited superior discrimination and calibration performance, while the join fusion achieved a more balanced distribution of the predicted probabilities. Discrimination performance did not significantly differ between strategies. Late fusion, though less technically complex, lacks crucial inter-modality interactions for NTCP modelling. In contrast, joint fusion, despite its complexity, resulted in a single network training process which included intra- and inter-modality interactions in its model parameter optimisation.Significance.This study is a pioneering effort in comparing different strategies for including image data into DL-based NTCP models in combination with lower dimensional data such as clinical variables. The discrimination performance of such multi-modality NTCP models and the choice of fusion strategy will depend on the distribution and quality of both types of data. Multiple data fusion strategies should be compared and reported in multi-modality NTCP modelling using DL.

CECS-CLIP: Fusing Domain Knowledge for Rare Wildlife Detection Model.

Accurate and efficient wildlife monitoring is essential for conservation efforts. Traditional image-based methods often struggle to detect small, occluded, or camouflaged animals due to the challenges posed by complex natural environments. To overcome these limitations, an innovative multimodal target detection framework is proposed in this study, which integrates textual information from an animal knowledge base as supplementary features to enhance detection performance. First, a concept enhancement module was developed, employing a cross-attention mechanism to fuse features based on the correlation between textual and image features, thereby obtaining enhanced image features. Secondly, a feature normalization module was developed, amplifying cosine similarity and introducing learnable parameters to continuously weight and transform image features, further enhancing their expressive power in the feature space. Rigorous experimental validation on a specialized dataset provided by the research team at Northwest A&F University demonstrates that our multimodal model achieved a 0.3% improvement in precision over single-modal methods. Compared to existing multimodal target detection algorithms, this model achieved at least a 25% improvement in AP and excelled in detecting small targets of certain species, significantly surpassing existing multimodal target detection model benchmarks. This study offers a multimodal target detection model integrating textual and image information for the conservation of rare and endangered wildlife, providing strong evidence and new perspectives for research in this field.