Joint Classification Research Articles

Asymmetric Feature Fusion Network for Hyperspectral and SAR Image Classification.

Joint classification using multisource remote sensing data for Earth observation is promising but challenging. Due to the gap of imaging mechanism and imbalanced information between multisource data, integrating the complementary merits for interpretation is still full of difficulties. In this article, a classification method based on asymmetric feature fusion, named asymmetric feature fusion network (AsyFFNet), is proposed. First, the weight-share residual blocks are utilized for feature extraction while keeping separate batch normalization (BN) layers. In the training phase, redundancy of the current channel is self-determined by the scaling factors in BN, which is replaced by another channel when the scaling factor is less than a threshold. To eliminate unnecessary channels and improve the generalization, a sparse constraint is imposed on partial scaling factors. Besides, a feature calibration module is designed to exploit the spatial dependence of multisource features, so that the discrimination capability is enhanced. Experimental results on the three datasets demonstrate that the proposed AsyFFNet significantly outperforms other competitive approaches.

Comparison of Sacroiliitis Grade Readings on the Same Plain Radiographs by the Same Observer at Different Periods.

This study aimed to investigate whether there is a difference between the readings of plain sacroiliac radiographs of patients with sacroiliitis by the same observer. In the study, we included patients diagnosed with sacroiliitis through sacroiliac MRI who had undergone plain radiographs at our center between 2015 and 2022. The radiographic grading of patients was conducted by transferring their demographic and clinical information into a computerized environmentso that these details would not be identifiable. The plain radiographs were numbered, and the responses were graded as grade 0, 1, 2, 3, or 4 for the right and left sacroiliac joints. The next day, using the same procedure, the same clinician re-evaluated the same plain radiographs in a different order without viewing the previous responses. This method was employed to prevent bias. The results (kappa value) were evaluated (0.00-0.20: slight agreement, 0.21-0.40: fair agreement, 0.41-0.60: moderate agreement, 0.61-0.80: substantial agreement, 0.81-1.00: perfect agreement). The study population included 478 patients and 956 sacroiliac joints from plain radiographs, both on the right and left. Following the observer's classification of the sacroiliac joints into 0, 1, 2, 3, and 4, a moderate level of agreement was found in the second evaluation of the same observer a day later with the same grades (p<0.001, kappa: 0.576). When categorized as grade0-1 and grade2-4, there was moderate agreement (p<0.001, kappa: 0.519), and categorization into grades 0-2 and 3-4 showed substantial agreement (p<0.001, kappa: 0.715). Analyzing the categorization into grades 0-3 and grade 4 revealed a higher kappa value, indicating substantial agreement (p<0.001, kappa: 0.766). Intraobserver interpretation of radiographs may be more accurate than the interpretation of different specialists. While interpreting plain radiographs, we observed variability between adjacent grades but less variability between distant grades. However, these results need to be validated.

Effect of weight change on the association between overall and source of carbohydrate intake and risk of metabolic syndrome: Tehran lipid and glucose study

BackgroundIn this prospective cohort study, we aimed to evaluate the association between dietary carbohydrates, whole grains, refined grains, and simple sugar with the risk of metabolic syndrome (MetS) and assess the effect of weight change on these associations.MethodsThis study was conducted within the framework of the Tehran Lipid and Glucose Study (TLGS). We included 1915 healthy participants with complete demographic, anthropometric and dietary measurements, among whom 591 developed MetS during 8.9 years of follow-up. Intake of dietary carbohydrates, whole grains, refined grains, and simple sugar was assessed with a validated semi-quantitative food frequency questionnaire. Multivariable adjusted Cox regression was used to estimate hazard ratios (HRs) for MetS events across tertiles of dietary variables. Using joint classification, the effect of weight change on the association between dietary variables and risk of MetS was assessed by Cox regression.ResultsCarbohydrate intake was not associated with the risk of MetS in multivariable-adjusted models. Whole grains intake was inversely associated with the risk of MetS (HR: 0.78, CI: 0.63–0.98), while this association disappeared after adjustment for weight change. The risk of MetS increased by 11% (1.11, 1.09–1.14) for each 3% energy increment from simple sugar, and by 5% (1.05, 1.03–1.08) for each 1 serving/day increment in refined grains. Consumption of refined grains increased the risk of MetS; an effect modification of this association was found by weight change. Among subjects with weight loss, intake of refined grains < median intake decreased the risk of MetS (0.59, 0.41–0.87). However, consumption of refined grains ≥ median intake increased the risk of MetS in individuals with weight gain (1.47, 1.08–2.01). Simple sugar was positively associated with an increased risk of MetS, after adjustment for weight change (3.00, 2.37–3.82). In joint classification, intake of simple sugar greater than median intake increased the risk of MetS, independent of weight change.ConclusionOur findings suggest an effect modification by weight change on the association between carbohydrates, and refined grains intake and the risk of MetS. Weight loss along with lower consumption of carbohydrates, and refined grains reduced the risk of MetS. However, simple sugar intake, regardless of weight change, was associated with an increased risk of MetS.

CloudY-Net: A Deep Convolutional Neural Network Architecture for Joint Segmentation and Classification of Ground-Based Cloud Images

Ground-based cloud images contain a wealth of cloud information and are an important part of meteorological research. However, in practice, ground cloud images must be segmented and classified to obtain the cloud volume, cloud type and cloud coverage. Existing methods ignore the relationship between cloud segmentation and classification, and usually only one of these is studied. Accordingly, our paper proposes a novel method for the joint classification and segmentation of cloud images, called CloudY-Net. Compared to the basic Y-Net framework, which extracts feature maps from the central layer, we extract feature maps from four different layers to obtain more useful information to improve the classification accuracy. These feature maps are combined to produce a feature vector to train the classifier. Additionally, the multi-head self-attention mechanism is implemented during the fusion process to enhance the information interaction among features further. A new module called Cloud Mixture-of-Experts (C-MoE) is proposed to enable the weights of each feature layer to be automatically learned by the model, thus improving the quality of the fused feature representation. Correspondingly, experiments are conducted on the open multi-modal ground-based cloud dataset (MGCD). The results demonstrate that the proposed model significantly improves the classification accuracy compared to classical networks and state-of-the-art algorithms, with classification accuracy of 88.58%. In addition, we annotate 4000 images in the MGCD for cloud segmentation and produce a cloud segmentation dataset called MGCD-Seg. Then, we obtain a 96.55 mIoU on MGCD-Seg, validating the efficacy of our method in ground-based cloud imagery segmentation and classification.

Enhanced Lightweight CNN Using Joint Classification With Averaging Probability for sEMG-Based Subject-Independent Hand Gesture Recognition

Enhanced Lightweight CNN Using Joint Classification With Averaging Probability for sEMG-Based Subject-Independent Hand Gesture Recognition

A novel insight towards classification of joints

A novel insight towards classification of joints

Symmetry-Based Fusion Algorithm for Bone Age Detection with YOLOv5 and ResNet34

Bone age is the chronological age of human bones, which serves as a key indicator of the maturity of bone development and can more objectively reflect the extent of human growth and development. The prevalent viewpoint and research development direction now favor the employment of deep learning-based bone age detection algorithms to determine bone age. Although bone age detection accuracy has increased when compared to more established methods, more work needs to be conducted to raise it because bone age detection is primarily used in clinical medicine, forensic identification, and other critical and rigorous fields. Due to the symmetry of human hand bones, bone age detection can be performed on either the left hand or the right hand, and the results are the same. In other words, the bone age detection results of both hands are universal. In this regard, the left hand is chosen as the target of bone age detection in this paper. To accomplish this, the You Only Look Once-v5 (YOLOv5) and Residual Network-34 (ResNet34) integration techniques are combined in this paper to create an innovative bone age detection model (YARN), which is then combined with the RUS-CHN scoring method that applies to Chinese adolescent children to comprehensively assess bone age at multiple levels. In this study, the images in the hand bone dataset are first preprocessed with number enhancement, then YOLOv5 is used to train the hand bone dataset to identify and filter out the main 13 joints in the hand bone, and finally, ResNet34 is used to complete the classification of local joints and achieve the determination of the developmental level of the detected region, followed by the calculation of the bone age by combining with the RUS-CHN method. The bone age detection model based on YOLOv5 and ResNet34 can significantly improve the accuracy and efficiency of bone age detection, and the model has significant advantages in the deep feature extraction of key regions of hand bone joints, which can efficiently complete the task of bone age detection. This was discovered through experiments on the public dataset of Flying Paddle AI Studio.

Invivo analysis of the circles measurement supports its use in evaluating acromioclavicular joint dislocations.

Classification and treatment of acromioclavicular joint (ACJ) dislocations according to the Rockwood classification is controversial. The "circles measurement" on Alexander views was proposed to enable a clear assessment of displacement in ACJ dislocations. However, the method and its ABC classification were introduced on a Sawbones model based on exemplary Rockwood scenarios without soft tissue. This is the first invivo study to investigate the circles measurement. We aimed to compare this new measurement method with the Rockwood classification and the previously described semiquantitative degree of dynamic horizontal translation (DHT). A total of 100 consecutive patients (87 men and 13 women) with acute ACJ dislocations between 2017 and 2020 were included retrospectively. The mean age was 41 years (range, 18-71 years). ACJ dislocations on panoramic stress views were classified according to Rockwood type (type II, n = 8; type IIIA, n = 9; type IIIB, n = 24; type IV, n = 7; and type V, n = 52). On Alexander views, where the hand of the affected arm rested on the contralateral shoulder, the circles measurement and the semiquantitative degree of DHT (none, n=6; partial, n=15; or complete, n=79) were assessed. Convergent and discriminant validity of the circles measurement (including its ABC classification according to displacement) with the coracoclavicular distance and Rockwood types, as well as the semiquantitative degree of DHT, was tested. The circles measurement showed a strong correlation with the coracoclavicular distance according to the Rockwood classification (r=0.66, P<.001) and differentiated between Rockwood types according to the ABC classification, including types IIIA and IIIB. The circles measurement correlated with the semiquantitative method of assessing DHT (r=0.61, P<.001). Measurement values were smaller in cases without DHT than in cases with partial DHT (P=.008). Cases with complete DHT had respectively larger measurement values (P<.001). In this first invivo study, the circles measurement allowed differentiation between Rockwood types according to the ABC classification in acute ACJ dislocations with a single measurement and correlated with the semiquantitative degree of DHT. On the basis of these validations of the circles measurement, its use for evaluating ACJ dislocations is recommended.

Visible Joint Classification and Temporal Segment Matching based 3D Pose Refinement for Volleyball Receive Analysis

Receive action pose analysis is very meaningful in volleyball games for training and strategy. Because receive action is lack high-quality labeled data sets and with problems like occlusion, body overlap, and abnormal pose. Conventional work fails to obtain high accurate pose results. This paper proposes visible joint refinement and receive action template matching for volleyball receive action analysis. Firstly, the visible joint is using pixel color feature and the potential constraint features between space and joints to classify the visible joint and refine the error visible joint. Secondly, the template is to realize the pose segment refinement at 3D level by matching. It is based on a multi-view system for a real volleyball competition scene. The dataset video is from the Game of 2014 Japan Inter High School of Men Volleyball. The experiment result achieves 95.33 %, 96.92 %, and 98.43 % success rate at the 30 mm, 50 mm, and 70 mm error ranges.

Classification of temporomandibular joint osteoarthritis on cone beam computed tomography images using artificial intelligence system.

The use of artificial intelligence has many advantages, especially in the field of oral and maxillofacial radiology. Early diagnosis of temporomandibular joint osteoarthritis by artificial intelligence may improve prognosis. The aim of this study is to perform the classification of temporomandibular joint (TMJ) osteoarthritis and TMJ segmentation on cone beam computed tomography (CBCT) sagittal images with artificial intelligence. In this study, the success of YOLOv5 architecture, an artificial intelligence model, in TMJ segmentation and osteoarthritis classification was evaluated on 2000 sagittal sections (500 healthy, 500 erosion, 500 osteophyte, 500 flattening images) obtained from CBCT DICOM images of 290 patients. The sensitivity, precision and F1 scores of the model for TMJ osteoarthritis classification are 1, 0.7678 and 0.8686, respectively. The accuracy value for classification is 0.7678. The prediction values of the classification model are 88% for healthy joints, 70% for flattened joints, 95% for joints with erosion and 86% for joints with osteophytes. The sensitivity, precision and F1 score of the YOLOv5 model for TMJ segmentation are 1, 0.9953 and 0.9976, respectively. The AUC value of the model for TMJ segmentation is 0.9723. In addition, the accuracy value of the model for TMJ segmentation was found to be 0.9953. Artificial intelligence model applied in this study can be a support method that will save time and convenience for physicians in the diagnosis of the disease with successful results in TMJ segmentation and osteoarthritis classification.

Joint Classification of Hyperspectral and LiDAR Data Using Binary-Tree Transformer Network

The joint utilization of multi-source data is of great significance in geospatial observation applications, such as urban planning, disaster assessment, and military applications. However, this approach is confronted with challenges including inconsistent data structures, irrelevant physical properties, scarce training data, insufficient utilization of information and an imperfect feature fusion method. Therefore, this paper proposes a novel binary-tree Transformer network (BTRF-Net), which is used to fuse heterogeneous information and utilize complementarity among multi-source remote sensing data to enhance the joint classification performance of hyperspectral image (HSI) and light detection and ranging (LiDAR) data. Firstly, a hyperspectral network (HSI-Net) is employed to extract spectral and spatial features of hyperspectral images, while the elevation information of LiDAR data is extracted using the LiDAR network (LiDAR-Net). Secondly, a multi-source transformer complementor (MSTC) is designed that utilizes the complementarity and cooperation among multi-modal feature information in remote sensing images to better capture their correlation. The multi-head complementarity attention mechanism (MHCA) within this complementor can effectively capture global features and local texture information of images, hence achieving full feature fusion. Then, to fully obtain feature information of multi-source remote sensing images, this paper designs a complete binary tree structure, binary feature search tree (BFST), which fuses multi-modal features at different network levels to obtain multiple image features with stronger representation abilities, effectively enhancing the stability and robustness of the network. Finally, several groups of experiments are designed to compare and analyze the proposed BTRF-Net with traditional methods and several advanced deep learning networks using two datasets: Houston and Trento. The results show that the proposed network outperforms other state-of-the-art methods even with small training samples.

Copper concentrate dual-band joint classification using reflectance hyperspectral images in the VIS-NIR and SWIR bands.

A study on the classification of copper concentrates relevant to the copper refining industry is performed by means of reflectance hyperspectral images in the visible and near infrared (VIS-NIR) bands (400-1000nm) and in the short-wave infrared (SWIR) (900-1700nm) band. A total of 82 copper concentrate samples were press compacted into 13-mm-diameter pellets, and their mineralogical composition was characterized via quantitative evaluation of minerals and scanning electron microscopy. The most representative minerals contained in these pellets are bornite, chalcopyrite, covelline, enargite, and pyrite. Three databases (VIS-NIR, SWIR, and VIS-NIR-SWIR) containing a collection of average reflectance spectra computed from 9×9p i x e l neighborhoods in each pellet hyperspectral image are compiled to train the classification models. The classification models tested in this work are a linear discriminant classifier and two non-linear classifiers, a quadratic discriminant classifier, and a fine K-nearest neighbor classifier (FKNNC). The results obtained show that the joint use of VIS-NIR and SWIR bands allows for the accurate classification of similar copper concentrates that contain only minor differences in their mineralogical composition. Specifically, among the three tested classification models, the FKNNC performs the best in terms of overall classification accuracy, achieving 93.4% accuracy in the test set when only VIS-NIR data are used to construct the classification model, up to 80.5% using only SWIR data, and up to 97.6% using both VIS-NIR and SWIR bands together.

Federated Multi-Task Learning for Joint Diagnosis of Multiple Mental Disorders on MRI Scans.

Deep learning (DL) techniques have been introduced to assist doctors in the interpretation of medical images by detecting image-derived phenotype abnormality. Yet the privacy-preserving policy of medical images disables the effective training of DL model using sufficiently large datasets. As a decentralized computing paradigm to address this issue, federated learning (FL) allows the training process to occur in individual institutions with local datasets, and then aggregates the resultant weights without risk of privacy leakage. We propose an effective federated multi-task learning (MTL) framework to jointly identify multiple related mental disorders based on functional magnetic resonance imaging data. A federated contrastive learning-based feature extractor is developed to extract high-level features across client models. To ease the optimization conflicts of updating shared parameters in MTL, we present a federated multi-gate mixture of expert classifier for the joint classification. The proposed framework also provides practical modules, including personalized model learning, privacy protection, and federated biomarker interpretation. On real-world datasets, the proposed framework achieves robust diagnosis accuracies of 69.48 ± 1.6%, 71.44 ± 3.2%, and 83.29 ± 3.2% in autism spectrum disorder, attention deficit/hyperactivity disorder, and schizophrenia, respectively. The proposed framework can effectively ease the domain shift between clients via federated MTL. The current work provides insights into exploiting the advantageous knowledge shared in related mental disorders for improving the generalization capability of computer-aided detection approaches.

Classification system for inter- and intra-module joints in non-sway steel MiC structures

The lack of guidelines for determining the rigidity of inter- and intra-module joints in corner-supported steel modular integrated construction (MiC) buildings can lead to unsafe or over-conservative designs. To address this issue, this study proposes a classification system for non-sway steel MiC buildings that covers both inter- and intra-module joints. First, the semi-rigidity of the joint regions in steel MiC buildings was clarified and the necessary degrees of freedom for joint classification were determined. Then a minimum substructure including one joint region was extracted from a steel MiC building, and its carrying capacity was determined as a simplified analytical model that considering the stiffnesses of both the inter- and intra-module joints. The simplified model was adopted to derive explicit formulas for the boundary limits of rigid/non-rigid inter- and intra-module joints, based on the criterion that the carrying capacity of the substructure with finite joint stiffnesses is 95% of that with rigid properties. Finally, the feasibility of the proposed classification system was verified by applying it to prototype MiC structures.

PIMA: Parameter-Shared Intelligent Media Analytics Framework for Low Resource Languages

Media analysis (MA) is an evolving area of research in the field of text mining and an important research area for intelligent media analytics. The fundamental purpose of MA is to obtain valuable insights that help to improve many different areas of business, and ultimately customer experience, through the computational treatment of opinions, sentiments, and subjectivity on mostly highly subjective text types. These texts can come from social media, the internet, and news articles with clearly defined and unique targets. Additionally, MA-related fields include emotion, irony, and hate speech detection, which are usually tackled independently from one another without leveraging the contextual similarity between them, mainly attributed to the lack of annotated datasets. In this paper, we present a unified framework to the complete intelligent media analysis, where we propose a shared parameter layer architecture with a joint learning approach that takes advantage of each separate task for the classification of sentiments, emotions, irony, and hate speech in texts. The proposed approach was evaluated on Greek expert-annotated texts from social media posts, news articles, and internet articles such as blog posts and opinion pieces. The results show that this joint classification approach improves the classification effectiveness of each task in terms of the micro-averaged F1-score.

ViViD: View Prediction of Online Video Through Deep Neural Network-Based Analysis of Subjective Video Attributes

Popularity of a video in an online platform may be defined by its number of views. The total view count of a video may change throughout its presence in an online platform. However, in most cases the view count tends to saturate after a certain time. We propose a method to predict the total view of a video at saturation. We have modeled the task of finding the view count of a video at saturation as a joint classification and regression problem, which is solved via a deep neural network. The network has a classification and a regression head. The classification head decides the view band among a set of available bands, whereas the regression head outputs a tolerance view count within each band. We consider four video attributes as the input to the network, namely, the thumbnail associated with the video, the title, the audio and the video itself for the view prediction task. The attributes are fused in a hierarchical fashion in the deep neural network. We propose a custom mismatch loss function and a penalty loss function for the joint training of the classification and regression heads of the network. Experimental results show that our method is 6.47% better in view prediction than the competitive methods.

A Novel Dual-Encoder Model for Hyperspectral and LiDAR Joint Classification via Contrastive Learning

Deep-learning-based multi-sensor hyperspectral image classification algorithms can automatically acquire the advanced features of multiple sensor images, enabling the classification model to better characterize the data and improve the classification accuracy. However, the currently available classification methods for feature representation in multi-sensor remote sensing data in their respective domains do not focus on the existence of bottlenecks in heterogeneous feature fusion due to different sensors. This problem directly limits the final collaborative classification performance. In this paper, to address the bottleneck problem of joint classification due to the difference in heterogeneous features, we innovatively combine self-supervised comparative learning while designing a robust and discriminative feature extraction network for multi-sensor data, using spectral–spatial information from hyperspectral images (HSIs) and elevation information from LiDAR. The advantages of multi-sensor data are realized. The dual encoders of the hyperspectral encoder by the ConvNeXt network (ConvNeXt-HSI) and the LiDAR encoder by Octave Convolution (OctaveConv-LiDAR) are also used. The adequate feature representation of spectral–spatial features and depth information obtained from different sensors is performed for the joint classification of hyperspectral images and LiDAR data. The multi-sensor joint classification performance of both HSI and LiDAR sensors is greatly improved. Finally, on the Houston2013 dataset and the Trento dataset, we demonstrate through a series of experiments that the dual-encoder model for hyperspectral and LiDAR joint classification via contrastive learning achieves state-of-the-art classification performance.

Fine-Grained Classification of Solder Joints With α-Skew Jensen–Shannon Divergence

Solder joint inspection (SJI) is a critical process in the production of printed circuit boards (PCB). Detection of solder errors during SJI is quite challenging as the solder joints have very small sizes and can take various shapes. In this study, we first show that solders have low feature diversity, and that the SJI can be carried out as a fine-grained image classification task which focuses on hard-to-distinguish object classes. To improve the fine-grained classification accuracy, penalizing confident model predictions by maximizing entropy was found useful in the literature. Inline with this information, we propose using the α-skew Jensen-Shannon divergence (α-JS) for penalizing the confidence in model predictions. We compare the α-JS regularization with both existing entropy-regularization based methods and the methods based on attention mechanism, segmentation techniques, transformer models, and specific loss functions for fine-grained image classification tasks. We show that the proposed approach achieves the highest F1-score and competitive accuracy for different models in the fine-grained solder joint classification task. Finally, we visualize the activation maps and show that with entropy-regularization, more precise class-discriminative regions are localized, which are also more resilient to noise. The code is available at https://github.com/furkanulger/SJI-entropy-reg.

Predicting Treatment Responses in Patients With Osteoarthritis: Results From Two Phase III Tanezumab Randomized Clinical Trials.

Prediction of treatment responses is essential to move forward translational science. Our question was to identify patient-based variables that predicted responses to treatments. We conducted secondary analyses on pooled data from two randomized phase III clinical trials (NCT02697773 and NCT02709486) conducted in participants with moderate to severe osteoarthritis randomized to subcutaneous placebo (n=514) or tanezumab 2.5 mg (n=514). We used gradient boosted regression trees to identify variables that predicted Western Ontario and McMaster Universities Osteoarthritis Index (WOMAC) Pain subscale scores at Week 16 and marginal plots to determine the directional relationship between each variable category and responses to placebo or tanezumab within the models. We also used Virtual Twins models to identify potential subgroups of response to the active treatment vs. placebo. We found that responses to placebo were predicted by baseline WOMAC Physical Function, baseline WOMAC Pain, the radiographic classification of the index joint, and the standard deviation of diary pain scores at baseline. In contrast, baseline WOMAC Pain along with failure of prior medications, duration of disease, and standard deviation of diary pain scores at baseline were predictive of tanezumab responses as expressed by the WOMAC Pain scores at Week 16. Those who responded to tanezumab vs. placebo were identified based on the radiographic classification of the index joint and either age or smoking status. These secondary-data analyses identified distinct and common patient-based variables to predict response to placebo or tanezumab. These findings will inform the design of future clinical trials, helping to move forward clinical pharmacology and translational science.

Attention Fusion of Transformer-Based and Scale-Based Method for Hyperspectral and LiDAR Joint Classification

In recent years, there have been many multimodal works in the field of remote sensing, and most of them have achieved good results in the task of land-cover classification. However, multi-scale information is seldom considered in the multi-modal fusion process. Secondly, the multimodal fusion task rarely considers the application of attention mechanism, resulting in a weak representation of the fused feature. In order to better use the multimodal data and reduce the losses caused by the fusion of different modalities, we proposed a TRMSF (Transformer and Multi-scale fusion) network for land-cover classification based on HSI (hyperspectral images) and LiDAR (Light Detection and Ranging) images joint classification. The network enhances multimodal information fusion ability by the method of attention mechanism from Transformer and enhancement using multi-scale information to fuse features from different modal structures. The network consists of three parts: multi-scale attention enhancement module (MSAE), multimodality fusion module (MMF) and multi-output module (MOM). MSAE enhances the ability of feature representation from extracting different multi-scale features of HSI, which are used to fuse with LiDAR feature, respectively. MMF integrates the data of different modalities through attention mechanism, thereby reducing the loss caused by the data fusion of different modal structures. MOM optimizes the network by controlling different outputs and enhances the stability of the results. The experimental results show that the proposed network is effective in multimodality joint classification.