Late Fusion Methods Research Articles

A Comparative approach of Depression Detection: A state of Art

Depression is an unconscious state of mind; it directly affects the mental health condition of patients, so it is required to accurately detect with a certain time to prevent severe depression. Depression detection with traditional approaches includes the patient's interviews and self-reporting to doctors or psychotherapists, but this approach faces challenges like time consumption and subjectivity. Another approach is to detect depression by using some computational techniques, including ‘machine learning (ML)’, ‘deep learning’, ‘natural language processing’, and also some wearable technologies or devices. The study compares the hybrid ensemble model and the late fusion method with other classifiers that are used for depression detection based on various performance parameters, like ‘accuracy’, classification report, ‘mean squared error’, and R2 score , etc. This study also discusses some ethical terms, such as the privacy of patient data and the risk of misdiagnosis. The study highlights the multimodal approach of various modality to build an effective and personalized system for depression detection.

Multimodal Sentiment Analysis for the Malay Language: New Corpus using CNN-based Framework

Sentiment analysis in the Malay language has traditionally focused on text-based data. Malay is the native language of Malaysia and other surrounding countries. While text-based sentiment analysis has shown good performance, it often lacks accuracy due to the absence of the speaker's affective state and intentions, which can lead to misinterpretations. Multimodal sentiment analysis addresses these shortcomings and has demonstrated improved performance in various prediction tasks. Unfortunately, there has been little research in this area for the Malay language, due to a lack of corpus and baseline studies. This paper introduces a new Malay Multimodal Sentiment Corpus, ‘MyMSC’, with annotations at both the multimodal and unimodal text levels. It contains 1208 segments covering political and social topics. The corpus development processes are described in detail, along with the necessary guidelines and considerations. This paper proposes a CNN-based framework with a late fusion method as the baseline model. Experiments with the proposed model demonstrate that the multimodal approach (F1 score = 0.77) outperforms the unimodal approach (F1 score = 0.68), validating the contribution of multimodality to classification performance. The differences between the two types of annotations and their impact are further elaborated. The full corpus is available at https://github.com/serenaroseaini/MyMSC .

Histopathology in focus: a review on explainable multi-modal approaches for breast cancer diagnosis.

Precision and timeliness in breast cancer detection are paramount for improving patient outcomes. Traditional diagnostic methods have predominantly relied on unimodal approaches, but recent advancements in medical data analytics have enabled the integration of diverse data sources beyond conventional imaging techniques. This review critically examines the transformative potential of integrating histopathology images with genomic data, clinical records, and patient histories to enhance diagnostic accuracy and comprehensiveness in multi-modal diagnostic techniques. It explores early, intermediate, and late fusion methods, as well as advanced deep multimodal fusion techniques, including encoder-decoder architectures, attention-based mechanisms, and graph neural networks. An overview of recent advancements in multimodal tasks such as Visual Question Answering (VQA), report generation, semantic segmentation, and cross-modal retrieval is provided, highlighting the utilization of generative AI and visual language models. Additionally, the review delves into the role of Explainable Artificial Intelligence (XAI) in elucidating the decision-making processes of sophisticated diagnostic algorithms, emphasizing the critical need for transparency and interpretability. By showcasing the importance of explainability, we demonstrate how XAI methods, including Grad-CAM, SHAP, LIME, trainable attention, and image captioning, enhance diagnostic precision, strengthen clinician confidence, and foster patient engagement. The review also discusses the latest XAI developments, such as X-VARs, LeGrad, LangXAI, LVLM-Interpret, and ex-ILP, to demonstrate their potential utility in multimodal breast cancer detection, while identifying key research gaps and proposing future directions for advancing the field.

A multimodal cross-transformer-based model to predict mild cognitive impairment using speech, language and vision

Mild Cognitive Impairment (MCI) is an early stage of memory loss or other cognitive ability loss in individuals who maintain the ability to independently perform most activities of daily living. It is considered a transitional stage between normal cognitive stage and more severe cognitive declines like dementia or Alzheimer’s. Based on the reports from the National Institute of Aging (NIA), people with MCI are at a greater risk of developing dementia, thus it is of great importance to detect MCI at the earliest possible to mitigate the transformation of MCI to Alzheimer’s and dementia. Recent studies have harnessed Artificial Intelligence (AI) to develop automated methods to predict and detect MCI. The majority of the existing research is based on unimodal data (e.g., only speech or prosody), but recent studies have shown that multimodality leads to a more accurate prediction of MCI. However, effectively exploiting different modalities is still a big challenge due to the lack of efficient fusion methods. This study proposes a robust fusion architecture utilizing an embedding-level fusion via a co-attention mechanism to leverage multimodal data for MCI prediction. This approach addresses the limitations of early and late fusion methods, which often fail to preserve inter-modal relationships. Our embedding-level fusion aims to capture complementary information across modalities, enhancing predictive accuracy. We used the I-CONECT dataset, where a large number of semi-structured conversations via internet/webcam between participants aged 75+ years old and interviewers were recorded. We introduce a multimodal speech-language-vision Deep Learning-based method to differentiate MCI from Normal Cognition (NC). Our proposed architecture includes co-attention blocks to fuse three different modalities at the embedding level to find the potential interactions between speech (audio), language (transcribed speech), and vision (facial videos) within the cross-Transformer layer. Experimental results demonstrate that our fusion method achieves an average AUC of 85.3% in detecting MCI from NC, significantly outperforming unimodal (60.9%) and bimodal (76.3%) baseline models. This superior performance highlights the effectiveness of our model in capturing and utilizing the complementary information from multiple modalities, offering a more accurate and reliable approach for MCI prediction.

Multimodal AI Combining Clinical and Imaging Inputs Improves Prostate Cancer Detection.

Deep learning (DL) studies for the detection of clinically significant prostate cancer (csPCa) on magnetic resonance imaging (MRI) often overlook potentially relevant clinical parameters such as prostate-specific antigen, prostate volume, and age. This study explored the integration of clinical parameters and MRI-based DL to enhance diagnostic accuracy for csPCa on MRI. We retrospectively analyzed 932 biparametric prostate MRI examinations performed for suspected csPCa (ISUP ≥2) at 2 institutions. Each MRI scan was automatically analyzed by a previously developed DL model to detect and segment csPCa lesions. Three sets of features were extracted: DL lesion suspicion levels, clinical parameters (prostate-specific antigen, prostate volume, age), and MRI-based lesion volumes for all DL-detected lesions. Six multimodal artificial intelligence (AI) classifiers were trained for each combination of feature sets, employing both early (feature-level) and late (decision-level) information fusion methods. The diagnostic performance of each model was tested internally on 20% of center 1 data and externally on center 2 data (n = 529). Receiver operating characteristic comparisons determined the optimal feature combination and information fusion method and assessed the benefit of multimodal versus unimodal analysis. The optimal model performance was compared with a radiologist using PI-RADS. Internally, the multimodal AI integrating DL suspicion levels with clinical features via early fusion achieved the highest performance. Externally, it surpassed baselines using clinical parameters (0.77 vs 0.67 area under the curve [AUC], P < 0.001) and DL suspicion levels alone (AUC: 0.77 vs 0.70, P = 0.006). Early fusion outperformed late fusion in external data (0.77 vs 0.73 AUC, P = 0.005). No significant performance gaps were observed between multimodal AI and radiologist assessments (internal: 0.87 vs 0.88 AUC; external: 0.77 vs 0.75 AUC, both P > 0.05). Multimodal AI (combining DL suspicion levels and clinical parameters) outperforms clinical and MRI-only AI for csPCa detection. Early information fusion enhanced AI robustness in our multicenter setting. Incorporating lesion volumes did not enhance diagnostic efficacy.

A novel multimodal depression diagnosis approach utilizing a new hybrid fusion method

In recent years, research has found that the impact of depression status primarily lies in patients’ language expression and facial expressions. Furthermore, facial expressions and intonation in speech exhibit a natural coexistence, making facial and vocal information core recognition indicators in depression identification. It is imperative to explore the effective use of deep learning methods for multimodal depression detection. We have proposed a novel trilateral bimodal encoding model (MEN), attentional decision fusion (ADF), and feature extraction fusion strategy. We employed a hybrid fusion approach that combines early intra-modality fusion with late inter-modality fusion, for multimodal depression diagnosis. In the feature extraction fusion component, we combine different representations of the same modality before inputting them into the network for training, enhancing features relevant to depression in the data. Through our multimodal encoding network, we extract frame-level information using Convolutional Neural Networks (CNN) while considering long-term context information and dependencies with Bidirectional Long Short-Term Memory (BiLSTM). Finally, the three streams of information were effectively integrated through attention fusion representation in our Attention Decision Fusion module (ADF), for depression score regression prediction. Extensive experiments were conducted on two public datasets, AVEC2013 and AVEC2014. The average absolute error/ root mean squared error (MAE/RMSE) scores for predicting depression scores were 6.48/8.91 and 7.01/9.38, respectively. This demonstrated that our hybrid fusion method outperforms traditional early or late fusion methods in terms of performance.

Infant head and brain segmentation from magnetic resonance images using fusion-based deep learning strategies

Magnetic resonance (MR) imaging is widely used for assessing infant head and brain development and for diagnosing pathologies. The main goal of this work is the development of a segmentation framework to create patient-specific head and brain anatomical models from MR images for clinical evaluation. The proposed strategy consists of a fusion-based Deep Learning (DL) approach that combines the information of different image sequences within the MR acquisition protocol, including the axial T1w, sagittal T1w, and coronal T1w after contrast. These image sequences are used as input for different fusion encoder–decoder network architectures based on the well-established U-Net framework. Specifically, three different fusion strategies are proposed and evaluated, namely early, intermediate, and late fusion. In the early fusion approach, the images are integrated at the beginning of the encoder–decoder architecture. In the intermediate fusion strategy, each image sequence is processed by an independent encoder, and the resulting feature maps are then jointly processed by a single decoder. In the late fusion method, each image is individually processed by an encoder–decoder, and the resulting feature maps are then combined to generate the final segmentations. A clinical in-house dataset consisting of 19 MR scans was used and divided into training, validation, and testing sets, with 3 MR scans defined as a fixed validation set. For the remaining 16 MR scans, a cross-validation approach was adopted to assess the performance of the methods. The training and testing processes were carried out with a split ratio of 75% for the training set and 25% for the testing set. The results show that the early and intermediate fusion methodologies presented the better performance (Dice coefficient of 97.6 ± 1.5% and 97.3 ± 1.8% for the head and Dice of 94.5 ± 1.7% and 94.8 ± 1.8% for the brain, respectively), whereas the late fusion method generated slightly worst results (Dice of 95.5 ± 4.4% and 93.8 ± 3.1% for the head and brain, respectively). Nevertheless, the volumetric analysis showed that no statistically significant differences were found between the volumes of the models generated by all the segmentation strategies and the ground truths. Overall, the proposed frameworks demonstrate accurate segmentation results and prove to be feasible for anatomical model analysis in clinical practice.

Hybrid Uncertainty Calibration for Multimodal Sentiment Analysis

In open environments, multimodal sentiment analysis (MSA) often suffers from low-quality data and can be disrupted by noise, inherent defects, and outliers. In some cases, unreasonable multimodal fusion methods can perform worse than unimodal methods. Another challenge of MSA is effectively enabling the model to provide accurate prediction when it is confident and to indicate high uncertainty when its prediction is likely to be inaccurate. In this paper, we propose an uncertain-aware late fusion based on hybrid uncertainty calibration (ULF-HUC). Firstly, we conduct in-depth research on the issue of sentiment polarity distribution in MSA datasets, establishing a foundation for an uncertain-aware late fusion method, which facilitates organic fusion of modalities. Then, we propose a hybrid uncertainty calibration method based on evidential deep learning (EDL) that balances accuracy and uncertainty, supporting the reduction of uncertainty in each modality of the model. Finally, we add two common types of noise to validate the effectiveness of our proposed method. We evaluate our model on three publicly available MSA datasets (MVSA-Single, MVSA-Multiple, and MVSA-Single-Small). Our method outperforms state-of-the-art approaches in terms of accuracy, weighted F1 score, and expected uncertainty calibration error (UCE) metrics, proving the effectiveness of the proposed method.

Deep learning based automated left ventricle segmentation and flow quantification in 4D flow cardiac MRI

Background4D flow MRI enables assessment of cardiac function and intra-cardiac blood flow dynamics from a single acquisition. However, due to the poor contrast between the chambers and surrounding tissue, quantitative analysis relies on the segmentation derived from a registered cine MRI acquisition. This requires an additional acquisition and is prone to imperfect spatial and temporal inter-scan alignment. Therefore, in this work we developed and evaluated deep learning-based methods to segment the left ventricle (LV) from 4D flow MRI directly. MethodsWe compared five deep learning-based approaches with different network structures, data pre-processing and feature fusion methods. For the data pre-processing, the 4D flow MRI data was reformatted into a stack of short-axis view slices. Two feature fusion approaches were proposed to integrate the features from magnitude and velocity images. The networks were trained and evaluated on an in-house dataset of 101 subjects with 67,567 2D images and 3030 3D volumes. The performance was evaluated using various metrics including Dice, average surface distance (ASD), end-diastolic volume (EDV), end-systolic volume (ESV), LV ejection fraction (LVEF), LV blood flow kinetic energy (KE) and LV flow components. The Monte Carlo dropout method was used to assess the confidence and to describe the uncertainty area in the segmentation results. ResultsAmong the five models, the model combining 2D U-Net with late fusion method operating on short-axis reformatted 4D flow volumes achieved the best results with Dice of 84.52% and ASD of 3.14 mm. The best averaged absolute and relative error between manual and automated segmentation for EDV, ESV, LVEF and KE was 19.93 ml (10.39%), 17.38 ml (22.22%), 7.37% (13.93%) and 0.07 mJ (5.61%), respectively. Flow component results derived from automated segmentation showed high correlation and small average error compared to results derived from manual segmentation. ConclusionsDeep learning-based methods can achieve accurate automated LV segmentation and subsequent quantification of volumetric and hemodynamic LV parameters from 4D flow MRI without requiring an additional cine MRI acquisition.

Fusion of Environmental Sensors for Occupancy Detection in a Real Construction Site

Internet-of-Things systems are increasingly being installed in buildings to transform them into smart ones and to assist in the transition to a greener future. A common feature of smart buildings, whether commercial or residential, is environmental sensing that provides information about temperature, dust, and the general air quality of indoor spaces, assisting in achieving energy efficiency. Environmental sensors though, especially when combined, can also be used to detect occupancy in a space and to increase security and safety. The most popular methods for the combination of environmental sensor measurements are concatenation and neural networks that can conduct fusion in different levels. This work presents an evaluation of the performance of multiple late fusion methods in detecting occupancy from environmental sensors installed in a building during its construction and provides a comparison of the late fusion approaches with early fusion followed by ensemble classifiers. A novel weighted fusion method, suitable for imbalanced samples, is also tested. The data collected from the environmental sensors are provided as a public dataset.

Multimodal Depression Detection: Using Fusion Strategies with Smart Phone Usage and Audio-visual Behavior

The problem of detecting depression is multi-faceted because of variability in depressive symptoms caused by individual differences. The variations can be seen in historical information (like decreased physical activity etc.) and also in verbal/non-verbal behaviors (like lower pitch, downward eye gaze etc.). The primary goal of this research is to develop a novel classification system for diagnosing depression that considers both historical information and also verbal/non-verbal behaviors. For this purpose, we created a realworld multimodal dataset of depressed and non-depressed subjects with fourteen-day real-time smartphone usage records and audio-visual recordings. We extracted numerous features related to physiological/physical activity from smartphone usage records to capture historical information and features like pitch and eye gaze (verbal and non-verbal manifestations) from audio-visual clues. We experimented with early fusion using Decision trees classifier (along with several feature selection strategies) and Support Vector Machine (SVM) classifier with several late fusion methods. Then, we conducted a comparative study among both fusion strategies. Our findings showed that SVM classifier using late fusion strategy achieves best accuracy of 89%. In addition, a popular benchmarking multimodal dataset (DAIC-WOZ database) is used to further validate the effectiveness of our approach by fusing multi-faceted feature vectors for depression detection.

Disease Recognition in X-ray Images with Doctor Consultation-Inspired Model.

The application of chest X-ray imaging for early disease screening is attracting interest from the computer vision and deep learning community. To date, various deep learning models have been applied in X-ray image analysis. However, models perform inconsistently depending on the dataset. In this paper, we consider each individual model as a medical doctor. We then propose a doctor consultation-inspired method that fuses multiple models. In particular, we consider both early and late fusion mechanisms for consultation. The early fusion mechanism combines the deep learned features from multiple models, whereas the late fusion method combines the confidence scores of all individual models. Experiments on two X-ray imaging datasets demonstrate the superiority of the proposed method relative to baseline. The experimental results also show that early consultation consistently outperforms the late consultation mechanism in both benchmark datasets. In particular, the early doctor consultation-inspired model outperforms all individual models by a large margin, i.e., 3.03 and 1.86 in terms of accuracy in the UIT COVID-19 and chest X-ray datasets, respectively.

Hateful Memes Detection Based on Multi-Task Learning

With the popularity of posting memes on social platforms, the severe negative impact of hateful memes is growing. As existing detection models have lower detection accuracy than humans, hateful memes detection is still a challenge to statistical learning and artificial intelligence. This paper proposed a multi-task learning method consisting of a primary multimodal task and two unimodal auxiliary tasks to address this issue. We introduced a self-supervised generation strategy in auxiliary tasks to generate unimodal auxiliary labels automatically. Meanwhile, we used BERT and RESNET as the backbone for text and image classification, respectively, and then fusion them with a late fusion method. In the training phase, the backward guidance technique and the adaptive weight adjustment strategy were used to capture the consistency and variability between different modalities, numerically improving the hateful memes detection accuracy and the generalization and robustness of the model. The experiment conducted on the Facebook AI multimodal hateful memes dataset shows that the prediction accuracy of our model outperformed the comparing models.

DFMKE: A dual fusion multi-modal knowledge graph embedding framework for entity alignment

Entity alignment is critical for multiple knowledge graphs (KGs) integration. Although researchers have made significant efforts to explore the relational embeddings between different KGs, existing approaches may not describe multi-modal knowledge well in some tasks, e.g., entity alignment. In this paper, we propose DFMKE, a dual fusion multi-modal knowledge graph embedding framework, to address entity alignment. We first devise an early fusion method for fusing features of multi-modal entity representations of a KG. Simultaneously, multiple representations of various types of knowledge are generated independently by various techniques and fused by a low-rank multi-modal late fusion method. Finally, the outputs of early and late fusion methods are combined using a dual fusion scheme. DFMKE provides an ultimate fusion solution by leveraging the advantages of early and late fusion methods. Extensive experiments on two public datasets show that the DFMKE outperforms state-of-the-art methods by a significant margin achieving at least 10% more regard to Hits@n and MRR metrics.

Towards a Reliable and Rapid Automated Grading System in Facial Palsy Patients: Facial Palsy Surgery Meets Computer Science.

Background: Reliable, time- and cost-effective, and clinician-friendly diagnostic tools are cornerstones in facial palsy (FP) patient management. Different automated FP grading systems have been developed but revealed persisting downsides such as insufficient accuracy and cost-intensive hardware. We aimed to overcome these barriers and programmed an automated grading system for FP patients utilizing the House and Brackmann scale (HBS). Methods: Image datasets of 86 patients seen at the Department of Plastic, Hand, and Reconstructive Surgery at the University Hospital Regensburg, Germany, between June 2017 and May 2021, were used to train the neural network and evaluate its accuracy. Nine facial poses per patient were analyzed by the algorithm. Results: The algorithm showed an accuracy of 100%. Oversampling did not result in altered outcomes, while the direct form displayed superior accuracy levels when compared to the modular classification form (n = 86; 100% vs. 99%). The Early Fusion technique was linked to improved accuracy outcomes in comparison to the Late Fusion and sequential method (n = 86; 100% vs. 96% vs. 97%). Conclusions: Our automated FP grading system combines high-level accuracy with cost- and time-effectiveness. Our algorithm may accelerate the grading process in FP patients and facilitate the FP surgeon’s workflow.

A Hybrid Multimodal Data Fusion-Based Method for Identifying Gambling Websites

With the development of network technology, the number of gambling websites has grown dramatically, causing a threat to social stability. There are many machine learning-based methods are proposed to identify gambling websites by analyzing the URL, the text, and the images of the websites. Nevertheless, most of the existing methods ignore one important piece of information, i.e., the text within the website images. Only the visual features of images are extracted for detection, while the semantic features of texts on the images are ignored. However, these texts have key information clearly pointing to gambling websites, which can help us identify such websites more accurately. Therefore, how to fuse image and text multimodal data is a challenge that should be met.Motivated by this, in this paper, we propose a hybrid multimodal data fusion-based method for identifying gambling websites by extracting and fusing visual and semantic features of the website screenshots. First, we fine tune the pretrained ResNet34 model to train an image classifier and to extract visual features of webpage screenshots. Second, we extract textual content from webpage screenshots through the optical character recognition (OCR) technique. We use pretrained Word2Vec word vectors as the initial embedding layer and use Bi-LSTM to train a text classifier and extract semantic features of textual content on the screenshots. Third, we use self-attention to fuse the visual and semantic features and train a multimodal classifier. The prediction results of image, text, and multimodal classifiers are fused by the late fusion method to obtain the final prediction result. To demonstrate the effectiveness of the proposed method, we conduct experiments on the webpage screenshot dataset we collected. The experimental results indicate that OCR text on the webpage screenshots has strong semantic features and the proposed hybrid multimodal data fusion based method can effectively improve the performance in identifying gambling websites, with accuracy, precision, recall, and F1-score all over 99%.

Identification of Children's Tuberous Sclerosis Complex with Multiple-contrast MRI and 3D Convolutional Network.

Identifying rare tuberous sclerosis complex (TSC) children is valuable and crucial. Magnetic resonance imaging (MRI) is used for rare TSC diagnoses. In this work, T2w and FLAIR were combined as a new modality named FLAIR3 to maximize the contrast between TSC lesions and normal-appearing brain tissues. After that, for the first time, we propose to use two different 3D CNN combined with late fusion strategies to diagnose TSC. A total of 520 children were enrolled in the study, including 260 health and 260 TSC children. The experiments had shown that the FLAIR3 could effectively improve the conspicuity of TSC lesions and classification performance. And the results showed the proposed late fusion method can improve the classification performance and achieve the state-of-the-art performance of the AUC of 0.994 and the accuracy of 0.971, which could be treated as an effective computer-aided diagnostic tool to help clinical radiologists diagnose TSC children. Clinical Relevance- Our deep learning method can be a non-invasive, efficient, and reliable way to help clinical radiologists to identify TSC patients. FLAIR3 can provide clinicians with a new modality to accurately localize TSC lesions in TSC patients.

Prediction of Academic Formulaic Language based on Multi-feature Fusion

&lt;p&gt;Academic formulaic language is multi-word combinations with specific functions and semantics, which are important to improve the idiomaticity, fluency and logic of machine translation, intelligent question answering, automatic summarization, etc. In order to narrow the search range of the corpus and extract academic formulaic language more efficiently, this paper proposes a prediction model of academic formulaic language based on multi-feature fusion. The semantic features and part-of-speech features of the academic formulaic language are extracted separately, and then the late fusion method is used to learn multiple features and predict whether formulaic language is included in the sentence. Experimental results show that the late fusion method based on part-of-speech features and semantic features has the best predictive effect among the four fusion methods, which lays the foundation for further efficient recognition of academic formulaic language.&lt;/p&gt; &lt;p&gt;&amp;nbsp;&lt;/p&gt;

MultiJAF: Multi-modal joint entity alignment framework for multi-modal knowledge graph

Entity Alignment (EA) is a crucial task in knowledge fusion, which aims to link entities with the same real-world identity from different Knowledge Graphs (KGs). Existing methods have achieved satisfactory performance, however, they mainly focus on single modal KG, which is difficult to be effectively applied to multi-modal scenes. In this paper, we propose a Multi-modal Joint entity Alignment Framework (MultiJAF), which can effectively utilize the knowledge of various modalities. Concretely, we first learn the embeddings of different modalities, i.e., structure, attribute and image modalities. Next, we adopt an attention-based multi-modal fusion network to integrate these embeddings and use obtained joint embeddings to compute a joint embedding-based similarity matrix SJ. Moreover, we design a Numerical Process Module (NPM) to infer a similarity matrix SN according to the numerical information of entities. In the end, we utilize a simple late fusion method to ensemble two similarity matrices for the final alignment. In addition, to reduce the cost of labeling data, we propose a novel NPM-based unsupervised multi-modal EA method. Experimental results on two real-world datasets demonstrate the effectiveness of our proposed MultiJAF.

Multi-sensor fusion of sparse point clouds based on neuralnet works

The fusion of laser point cloud and visual image depends on the point cloud density and the target framing effect, the traditional laser point cloud processing for sparse point cloud clustering effect is poor, it is difficult to frame small objects as well as medium and long distance objects. Then the subsequent sensor fusion is easy to miss the recognition of obstacles. In this paper, we improve the frame selection method for sparse point clouds, firstly build a deep learning framework pointpillar, use pointpillar to frame the sparse laser point clouds, then spatially calibrate the lidar coordinate system and camera coordinate system, project the lidar point clouds to the camera image, improve the late fusion method, effectively use the detection results of single sensor, and finally The late-fusion is performed with the target detection results of the camera image to output the exact distance as well as the category of the target. Experiments show that compared with the recognition effect of the traditional fusion algorithm, the number of frames is increased by 6 and the missed recognition rate is reduced from 31.41% to 12.31%.