Multi-label Convolutional Neural Network Research Articles

Joint Inversion of Surface-Wave Dispersions and Receiver Functions Based on Deep Learning

Abstract We proposed a deep learning (DL) method to derive VS models from joint inversion of Rayleigh-wave dispersions and receiver functions, which is based on multilabel convolutional neural network and recurrent neural network. We used a spline-based approach to generate synthetic models instead of directly using existing models to build the training data set, which improves the generalization of the method. Unlike the traditional methods, which usually set a fixed VP/VS ratio, our method makes full use of the powerful data mining ability of DL to invert the VS models assuming different VP/VS ratios. A loss function is specially designed that focuses on key features of the model space, for example, the shape and depth of Moho. Synthetic tests demonstrate that the proposed method is accurate and fast. Application to the southeast margin of the Tibetan Plateau shows results consistent with the previous joint inversion with P constraints, indicating the proposed method is reliable and robust.

Multiple marine algae identification based on three-dimensional fluorescence spectroscopy and multi-label convolutional neural network

Accurate identification of algal populations plays a pivotal role in monitoring seawater quality. Fluorescence-based techniques are effective tools for quickly identifying different algae. However, multiple coexisting algae and their similar photosynthetic pigments can constrain the efficacy of fluorescence methods. This study introduces a multi-label classification model that combines a specific Excitation-Emission matric convolutional neural network (EEM-CNN) with three-dimensional (3D) fluorescence spectroscopy to detect single and mixed algal samples. Spectral data can be input directly into the model without transforming into images. Rectangular convolutional kernels and double convolutional layers are applied to enhance the extraction of balanced and comprehensive spectral features for accurate classification. A dataset comprising 3D fluorescence spectra from eight distinct algae species representing six different algal classes was obtained, preprocessed, and augmented to create input data for the classification model. The classification model was trained and validated using 4448 sets of test samples and 60 sets of test samples, resulting in an accuracy of 0.883 and an F1 score of 0.925. This model exhibited the highest recognition accuracy in both single and mixed algae samples, outperforming comparative methods such as ML-kNN and N-PLS-DA. Furthermore, the classification results were extended to three different algae species and mixed samples of skeletonema costatum to assess the impact of spectral similarity on multi-label classification performance. The developed classification models demonstrated robust performance across samples with varying concentrations and growth stages, highlighting CNN's potential as a promising tool for the precise identification of marine algae.

MLCNN‐COV: A multilabel convolutional neural network‐based framework to identify negative COVID medicine responses from the chemical three‐dimensional conformer

AbstractTo treat the novel COronaVIrus Disease (COVID), comparatively fewer medicines have been approved. Due to the global pandemic status of COVID, several medicines are being developed to treat patients. The modern COVID medicines development process has various challenges, including predicting and detecting hazardous COVID medicine responses. Moreover, correctly predicting harmful COVID medicine reactions is essential for health safety. Significant developments in computational models in medicine development can make it possible to identify adverse COVID medicine reactions. Since the beginning of the COVID pandemic, there has been significant demand for developing COVID medicines. Therefore, this paper presents the transfer‐learning methodology and a multilabel convolutional neural network for COVID (MLCNN‐COV) medicines development model to identify negative responses of COVID medicines. For analysis, a framework is proposed with five multilabel transfer‐learning models, namely, MobileNetv2, ResNet50, VGG19, DenseNet201, and Inceptionv3, and an MLCNN‐COV model is designed with an image augmentation (IA) technique and validated through experiments on the image of three‐dimensional chemical conformer of 17 number of COVID medicines. The RGB color channel is utilized to represent the feature of the image, and image features are extracted by employing the Convolution2D and MaxPooling2D layer. The findings of the current MLCNN‐COV are promising, and it can identify individual adverse reactions of medicines, with the accuracy ranging from 88.24% to 100%, which outperformed the transfer‐learning model's performance. It shows that three‐dimensional conformers adequately identify negative COVID medicine responses.

Palm bunch grading technique using a multi-input and multi-label convolutional neural network

Oil palm bunches, which are the raw materials for several oil palm products, are traded to a crude palm oil (CPO) factory or a middleman. Since high-oil-content palm bunches are desired to produce CPO, the factories and middlemen traditionally have several experts hired for grading FFBs. We have found that colour, shape and size of FFBs are three parameters that need to be considered for classification. Shape and colour data can be collected using RGB and IR sensors, while size data can be collected using IR and weight sensors. In this paper, we propose a deep learning-based model working with three sensors (RGB camera, infrared sensor, and a load cell) for classifying a palm bunch. The classifying framework of the algorithm is a multi-input and multi-label convolutional neural network. The multi-label classifies the 4 grades of FFBs (overripe, ripe, under ripe and unripe). Based on 1,575 images for 14 varieties of palm bunches collected from four trading sites, the sensors were installed, the algorithm was trained, validated, and tested with 70%, 20% and 10% of the overall datasets, respectively. The general images that are preprocessed after training model are applied and compared between original and preprocess images by Gradient-weighted Class Activation Mapping (Grad-CAM). The cross-entropy loss function was applied to investigate the probability error between the target and predicted label during the training process. Based on the testing datasets results, the proposed model achieves an accuracy of 90.26%, precision of 89.86%, recall of 89.54%, and an F1-score of 89.68%.

A convolutional neural network algorithm developed for shielded multi-isotope identification

Identifying radioisotopes can facilitate risk-informed management decisions in nuclear security and nuclear site decommissioning applications. An artificial neural network based algorithm capable of identifying multiple radioisotopes across a range of shielded environments is presented. Five isotopes were selected to create this algorithm. Small sets of single-isotope gamma spectra were collected using a sodium iodide detector in three different shielding configurations. Each of these sets of spectra was independently augmented by combining spectra to create multi-isotope spectra. The three sets were then combined to form a training set containing 38500 spectra. Four types of neural networks were explored, optimised, and compared. Based on the results of this comparison, a multi-label convolutional neural network with five hidden layers was chosen to test on a large set of 2132 spectra using sources of the same isotopes with different activities. These testing spectra were collected experimentally and independently of the training set. Subsets of spectra were taken from the testing set and used to assess the ability of the algorithm in four different ways: to identify multiple spectra simultaneously, to adapt to different shielding configurations, to explore the decrease in performance when analysing count-starved spectra, and to analyse spectra from a different NaI detector. The precision, recall, and F1 scores were calculated on all tests. Results show a high performance with F1 scores that are typically above 0.85. Recall values were generally higher than precision when analysing shielded spectra, indicating that the algorithm tends to produce more false positives than negatives.

A multi-label CNN model for the automatic detection and segmentation of gliomas using [18F]FET PET imaging.

The aim of this study was to develop a convolutional neural network (CNN) for the automatic detection and segmentation of gliomas using [18F]fluoroethyl-L-tyrosine ([18F]FET) PET. Ninety-three patients (84 in-house/7 external) who underwent a 20-40-min static [18F]FET PET scan were retrospectively included. Lesions and background regions were defined by two nuclear medicine physicians using the MIM software, such that delineations by one expert reader served as ground truth for training and testing the CNN model, while delineations by the second expert reader were used to evaluate inter-reader agreement. A multi-label CNN was developed to segment the lesion and background region while a single-label CNN was implemented for a lesion-only segmentation. Lesion detectability was evaluated by classifying [18F]FET PET scans as negative when no tumor was segmented and vice versa, while segmentation performance was assessed using the dice similarity coefficient (DSC) and segmented tumor volume. The quantitative accuracy was evaluated using the maximal and mean tumor to mean background uptake ratio (TBRmax/TBRmean). CNN models were trained and tested by a threefold cross-validation (CV) using the in-house data, while the external data was used for an independent evaluation to assess the generalizability of the two CNN models. Based on the threefold CV, the multi-label CNN model achieved 88.9% sensitivity and 96.5% precision for discriminating between positive and negative [18F]FET PET scans compared to a 35.3% sensitivity and 83.1% precision obtained with the single-label CNN model. In addition, the multi-label CNN allowed an accurate estimation of the maximal/mean lesion and mean background uptake, resulting in an accurate TBRmax/TBRmean estimation compared to a semi-automatic approach. In terms of lesion segmentation, the multi-label CNN model (DSC = 74.6 ± 23.1%) demonstrated equal performance as the single-label CNN model (DSC = 73.7 ± 23.2%) with tumor volumes estimated by the single-label and multi-label model (22.9 ± 23.6ml and 23.1 ± 24.3ml, respectively) closely approximating the tumor volumes estimated by the expert reader (24.1 ± 24.4ml). DSCs of both CNN models were in line with the DSCs by the second expert reader compared with the lesion segmentations by the first expert reader, while detection and segmentation performance of both CNN models as determined with the in-house data were confirmed by the independent evaluation using external data. The proposed multi-label CNN model detected positive [18F]FET PET scans with high sensitivity and precision. Once detected, an accurate tumor segmentation and estimation of background activity was achieved resulting in an automatic and accurate TBRmax/TBRmean estimation, such that user interaction and potential inter-reader variability can be minimized.

Transfer reinforcement learning method with multi-label learning for compound fault recognition

In complex working site, bearings used as the important part of machine, could simultaneously have faults on several positions. Consequently, multi-label learning approach considering fully the correlation between different faulted positions of bearings becomes the popular learning pattern. Deep reinforcement learning (DRL) combining the perception ability of deep learning and the decision-making ability of reinforcement learning, could be adapted to the compound fault diagnosis while having a strong ability extracting the fault feature from the raw data. However, DRL is difficult to converge and easily falls into the unstable training problem. Therefore, this paper integrates the feature extraction ability of DRL and the knowledge transfer ability of transfer learning (TL), and proposes the multi-label transfer reinforcement learning (ML-TRL). In detail, the proposed method utilizes the improved trust region policy optimization (TRPO) as the basic DRL framework and pre-trains the fixed convolutional networks of ML-TRL using the multi-label convolutional neural network method. In compound fault experiment, the final results demonstrate powerfully that the proposed method could have the higher accuracy than other multi-label learning methods. Hence, the proposed method is a remarkable alternative when recognizing the compound fault of bearings.

Multiple Ocular Disease Diagnosis Using Fundus Images Based on Multi-Label Deep Learning Classification

Designing computer-aided diagnosis (CAD) systems that can automatically detect ocular diseases (ODs) has become an active research field in the health domain. Although the human eye might have more than one OD simultaneously, most existing systems are designed to detect specific eye diseases. Therefore, it is crucial to develop new CAD systems that can detect multiple ODs simultaneously. This paper presents a novel multi-label convolutional neural network (ML-CNN) system based on ML classification (MLC) to diagnose various ODs from color fundus images. The proposed ML-CNN-based system consists of three main phases: the preprocessing phase, which includes normalization and augmentation using several transformation processes, the modeling phase, and the prediction phase. The proposed ML-CNN consists of three convolution (CONV) layers and one max pooling (MP) layer. Then, two CONV layers are performed, followed by one MP and dropout (DO). After that, one flatten layer is performed, followed by one fully connected (FC) layer. We added another DO once again, and finally, one FC layer with 45 nodes is performed. The system outputs the probabilities of all 45 diseases in each image. We validated the model by using cross-validation (CV) and measured the performance by five different metrics: accuracy (ACC), recall, precision, Dice similarity coefficient (DSC), and area under the curve (AUC). The results are 94.3%, 80%, 91.5%, 99%, and 96.7%, respectively. The comparisons with the existing built-in models, such as MobileNetV2, DenseNet201, SeResNext50, InceptionV3, and InceptionresNetv2, demonstrate the superiority of the proposed ML-CNN model.

Development and multimodal validation of a substance misuse algorithm for referral to treatment using artificial intelligence (SMART-AI): a retrospective deep learning study

SummaryBackgroundSubstance misuse is a heterogeneous and complex set of behavioural conditions that are highly prevalent in hospital settings and frequently co-occur. Few hospital-wide solutions exist to comprehensively and reliably identify these conditions to prioritise care and guide treatment. The aim of this study was to apply natural language processing (NLP) to clinical notes collected in the electronic health record (EHR) to accurately screen for substance misuse.MethodsThe model was trained and developed on a reference dataset derived from a hospital-wide programme at Rush University Medical Center (RUMC), Chicago, IL, USA, that used structured diagnostic interviews to manually screen admitted patients over 27 months (between Oct 1, 2017, and Dec 31, 2019; n=54 915). The Alcohol Use Disorder Identification Test and Drug Abuse Screening Tool served as reference standards. The first 24 h of notes in the EHR were mapped to standardised medical vocabulary and fed into single-label, multilabel, and multilabel with auxillary-task neural network models. Temporal validation of the model was done using data from the subsequent 12 months on a subset of RUMC patients (n=16 917). External validation was done using data from Loyola University Medical Center, Chicago, IL, USA between Jan 1, 2007, and Sept 30, 2017 (n=1991 adult patients). The primary outcome was discrimination for alcohol misuse, opioid misuse, or non-opioid drug misuse. Discrimination was assessed by the area under the receiver operating characteristic curve (AUROC). Calibration slope and intercept were measured with the unreliability index. Bias assessments were performed across demographic subgroups.FindingsThe model was trained on a cohort that had 3·5% misuse (n=1 921) with any type of substance. 220 (11%) of 1921 patients with substance misuse had more than one type of misuse. The multilabel convolutional neural network classifier had a mean AUROC of 0·97 (95% CI 0·96–0·98) during temporal validation for all types of substance misuse. The model was well calibrated and showed good face validity with model features containing explicit mentions of aberrant drug-taking behaviour. A false-negative rate of 0·18–0·19 and a false-positive rate of 0·03 between non-Hispanic Black and non-Hispanic White groups occurred. In external validation, the AUROCs for alcohol and opioid misuse were 0·88 (95% CI 0·86–0·90) and 0·94 (0·92–0·95), respectively.InterpretationWe developed a novel and accurate approach to leveraging the first 24 h of EHR notes for screening multiple types of substance misuse.FundingNational Institute On Drug Abuse, National Institutes of Health.

Optimized multi‐label convolutional neural network using modified genetic algorithm for popularity based personalized news recommendation system

AbstractClassification of label‐specific users' diversified interests and incorporating social media with news media to address popular news articles which is the most formidable task in popularity‐based personalized news recommendation systems (PPNRS). To bring personalization to PPNRS, many remarkable features have to be considered from users user profiles to classify their interest. In this article, 13,346 features per user considered to classify their interest for 15 labels using multi‐label convolutional neural network (MLCNN). The efficiency of MLCNN model highly depends on its architecture through the tuning of its hyperparameters. Generally, researchers manually designed a constant CNN architecture for every label and verified the effectiveness, but this leads to an additional complexity as well as large computational resources were consumed. Moreover, designing the structure for all 15 labels leads to an increase in the network structure exponentially with an increase in labels. Hence, in this manuscript, MLCNN architectures optimized by implementing a novel approach modified genetic algorithm (MGA) with the help of introducing four novel crossover operators to strengthen CNN performance for users interest classification. Further, for the recommendation process, the label‐specific news articles were clustered from social media Facebook and Twitter feeds, and then most popular news articles determined from clusters along with label‐specific breaking news articles rendered from news feeds concerning users' interest. In addition to that, the reliability of the news articles also validated for recommendation process. The experimental result precisely proves that the proposed approach MGA attained an accuracy of 89.64%, 90.56%, 90.41%, and 91.79% for classifying users label specific interest and label‐wise recommendation accuracy attained 93.3%, 90%, 90% from Twitter, Facebook, and also from Newsfeed respectively.

Simulating mixed land-use change under multi-label concept by integrating a convolutional neural network and cellular automata: a case study of Huizhou, China

ABSTRACT Cellular automata (CA) model is a useful tool for simulating spatiotemporal changes of land-use evolution. However, previous CA models have usually ignored the inhomogeneous and mixed land-use situations and simplified the land grid cell as one single land-use type. This study developed a multi-label (ML) convolutional neural network CA (ML-CNN-CA) model to simulate the complex evolution of mixed land uses. The multi-label learning strategy in the proposed model can formulize mixed land uses by labeling multiple land-use types to each grid cell. The transition rules of CA were modified to accommodate the multi-label learning strategy and mined by coupling a CNN network and three interactive mechanisms, including the neighborhood effect, adaptive inertia coefficient, and random factor. The proposed ML-CNN-CA model was applied and examined in a fast-developing city (Huizhou) in the China’s Pearl River Delta region for the mixed land-use simulation during 2009–2013 and 2013–2020. The ML-CNN-CA model with different CNN architectures and a previous artificial neural network (ANN)-based multi-label CA model were also implemented for comparisons. Results show the capability of the ML-CNN-CA model to simulate fine-scale mixed land-use changes with satisfactory performances. Specifically, the simulated mixed land-use patterns agree well with the actual land uses from the morphological perspective. Furthermore, the quantitative assessments demonstrate the performance of the proposed model, showing an accuracy value of 0.912 for 2009–2013 and 0.896 for 2013–2020, and a hamming loss value of 0.048 for 2009–2013 and 0.055 for 2013–2020. The comparisons also show the best performance of the ML-CNN-CA model with the VGG-based architecture and significant outperformance of the proposed model against the previous ANN-based CA model. Multiple sensitivity analyses were also conducted to investigate the uncertainty of the proposed model. The ML-CNN-CA model proposed in this study can provide a new tool for better simulation of fine-scale mixed land-use changes and is expected to help formulate urban planning guidelines and achieve sustainable urban development.

Control Chart Concurrent Pattern Classification Using Multi-Label Convolutional Neural Networks

The detection and identification of non-random patterns is an important task in statistical process control (SPC). When a non-random pattern appears on a control chart, it means that there are assignable causes which will gradually deteriorate the process quality. In addition to the study of a single pattern, many researchers have also studied concurrent non-random patterns. Although concurrent patterns have multiple characteristics from different basic patterns, most studies have treated them as a special pattern and used the multi-class classifier to perform the classification work. This study proposed a new method that uses a multi-label convolutional neural network to construct a classifier for concurrent patterns of a control chart. This study used data from previous studies to evaluate the effectiveness of the proposed method with appropriate multi-label classification metrics. The results of the study show that the recognition performance of multi-label convolutional neural network is better than traditional machine learning algorithms. This study also used real-world data to demonstrate the applicability of the proposed method to online monitoring. This study aids in the further realization of smart SPC.

Structural damage identification under nonstationary excitations through recurrence plot and multi-label convolutional neural network

Civil engineering structures inevitably suffer from nonstationary ambient excitations in practice, which make conventional damage identification methods relying on the stationary assumption ineffective. This study presents a novel method based on unthresholded assembled recurrence distance matrix (UARDM) and multi-label convolutional neural network (CNN) for structural damage identification under nonstationary excitations. UARDM is a new type of recurrence plot (RP) that is proposed to integrate information of multiple channels and dispense with the artificially selected threshold. It reveals intrinsic dynamic characteristics of the structure using its vibration responses from the perspective of global probabilistic autocorrelation. After that, CNN is applied to automatically extract damage-sensitive features of UARDMs and classify them for the identification of damage cases. Instead of the traditional single-label CNN model that labels each combination of damage location and level as an objective class, the multi-label CNN model is developed to decouple the identification processes of damage locations and levels in order to improve the identification accuracy and computational efficiency. It evaluates the damage level at each location through a sub-branch with an independent set of labels and detects the damage locations by fusing information of all the sub-branches. A comprehensive comparison was conducted among single-label and multi-label CNN models input with raw accelerations, unthresholded multivariate recurrence plots (UMRPs), unthresholded recurrence plots (URPs) and UARDMs through numerical simulation and experimental test. It was demonstrated that the proposed structural damage identification method based on UARDM and multi-label CNN was able to identify multiple damage locations and levels under various stationary and nonstationary excitations with higher accuracy, efficiency and robustness, and even able to detect multiple-damage cases that were not measured beforehand and involved in the training dataset.

Convolutional Neural Networks for Challenges in Automated Nuclide Identification.

Improvements in Radio-Isotope IDentification (RIID) algorithms have seen a resurgence in interest with the increased accessibility of machine learning models. Convolutional Neural Network (CNN)-based models have been developed to identify arbitrary mixtures of unstable nuclides from gamma spectra. In service of this, methods for the simulation and pre-processing of training data were also developed. The implementation of 1D multi-class, multi-label CNNs demonstrated good generalisation to real spectra with poor statistics and significant gain shifts. It is also shown that even basic CNN architectures prove reliable for RIID under the challenging conditions of heavy shielding and close source geometries, and may be extended to generalised solutions for pragmatic RIID.

Substance Misuse Algorithm for Referral to Treatment Using Artificial Intelligence (SMART-AI): Multi-Modal Validation with Interpretation and Bias Assessment

Background: Substance misuse is a heterogeneous and complex set of behavioral conditions that are highly prevalent in hospital settings and frequently co-occur. Few solutions exist to comprehensively and reliably identify these conditions hospital-wide to prioritize care and guide treatment. The aim is apply natural language processing (NLP) to admission notes in the electronic health record (EHR) to accurately screen for substance misuse. Methods: The reference dataset was derived from a hospital-wide program that used structured diagnostic interviews to manually screen admitted patients over 26 months (n=54,915). Temporal validation was provided over the subsequent 12 months (n=16,917) and external validation at a separate health system (n=1,991). The Alcohol Use Disorder Identification Test and Drug Abuse Screening Tool served as reference standards. The first 24 hours of notes in the EHR were mapped to standardized medical vocabulary and fed into neural network models. The primary outcome was discrimination for alcohol misuse, opioid misuse, or non-opioid drug misuse. Discrimination was assessed by the area under the receiver operating curve (AUROC). Findings: The model was trained on a cohort that had 3.5% (n=1,921) with any type of substance misuse. Nearly 11% of patients with substance misuse had more than one type of misuse. The multi-label convolutional neural network classifier had an average AUROC of 0.97 (95% CI: 0.96, 0.98) during temporal validation for all types of substance misuse. The model was well calibrated and demonstrated good face validity with model features containing explicit mentions of aberrant drug-taking behavior. The false-negative and false-positive rates were similar between non-Hispanic Black and non-Hispanic White groups. In external validation, the AUROC for alcohol and opioid misuse remained above 0.85. Interpretation: We developed a novel and accurate approach to leveraging the first 24 hours of EHR notes for screening multiple types of substance misuse. Funding Information: Research reported in this publication was supported by the National Institute On Drug Abuse of the National Institutes of Health under Award Numbers R01-DA051464 (MA), K23-AA024503 (MA), UL1-TR002389 (NK), KL2-TR002387 (NK), R01-DA041071 (NK), UG1-DA049467 (NK), R01-LM010090 (DD), R01-LM012973 (DD), K12-HS-026385 (HT), and R01-GM123193 (MMC). Declaration of Interests: Dr. Churpek has a patent pending (ARCD. P0535US.P2) for risk stratification algorithms for hospitalized patients, and has received research support from EarlySense (Tel Aviv, Israel). All other authors have nothing to declare. Ethics Approval Statement: This study was approved by the Institutional Review Board at RUMC and LUMC.

Fully Automatic Atrial Fibrosis Assessment Using a Multilabel Convolutional Neural Network.

Pathological atrial fibrosis is a major contributor to sustained atrial fibrillation. Currently, late gadolinium enhancement (LGE) scans provide the only noninvasive estimate of atrial fibrosis. However, widespread adoption of atrial LGE has been hindered partly by nonstandardized image processing techniques, which can be operator and algorithm dependent. Minimal validation and limited access to transparent software platforms have also exacerbated the problem. This study aims to estimate atrial fibrosis from cardiac magnetic resonance scans using a reproducible operator-independent fully automatic open-source end-to-end pipeline. A multilabel convolutional neural network was designed to accurately delineate atrial structures including the blood pool, pulmonary veins, and mitral valve. The output from the network removed the operator dependent steps in a reproducible pipeline and allowed for automated estimation of atrial fibrosis from LGE-cardiac magnetic resonance scans. The pipeline results were compared against manual fibrosis burdens, calculated using published thresholds: image intensity ratio 0.97, image intensity ratio 1.61, and mean blood pool signal +3.3 SD. We validated our methods on a large 3-dimensional LGE-cardiac magnetic resonance data set from 207 labeled scans. Automatic atrial segmentation achieved a 91% Dice score, compared with the mutual agreement of 85% in Dice seen in the interobserver analysis of operators. Intraclass correlation coefficients of the automatic pipeline with manually generated results were excellent and better than or equal to interobserver correlations for all 3 thresholds: 0.94 versus 0.88, 0.99 versus 0.99, 0.99 versus 0.96 for image intensity ratio 0.97, image intensity ratio 1.61, and +3.3 SD thresholds, respectively. Automatic analysis required 3 minutes per case on a standard workstation. The network and the analysis software are publicly available. Our pipeline provides a fully automatic estimation of fibrosis burden from LGE-cardiac magnetic resonance scans that is comparable to manual analysis. This removes one key source of variability in the measurement of atrial fibrosis.

Multi-level similarity learning for image-text retrieval

Image-text retrieval task has been a popular research topic and attracts a growing interest due to it bridges computer vision and natural language processing communities and involves two different modalities. Although a lot of methods have made a great progress in image-text task, it remains challenging because of the difficulty to learn the correspondence between two heterogeneous modalities. In this paper, we propose a multi-level representation learning for image-text retrieval task, which utilizes semantic-level, structural-level and contextual information to improve the quality of visual and textual representation. To utilize semantic-level information, we firstly extract the nouns, adjectives and number with high frequency as the semantic labels and adopt multi-label convolutional neural network framework to encode the semantic-level information. To explore the structure-level information of image-text pair, we firstly construct two graphs to encode the visual and textual information with respect to the corresponding modality and then, we apply graph matching with triplet loss to reduce the cross-modality discrepancy. To further improve the retrieval results, we utilize the contextual-level information from two modalities to refine the rank list and enhance the retrieval quality. Extensive experiments on Flickr30k and MSCOCO, which are two commonly datasets for image-text retrieval, have demonstrated the superiority of our proposed method.

Multi-label fault diagnosis of rolling bearing based on meta-learning

In practical applications, it is difficult to acquire sufficient fault samples for training deep learning fault diagnosis model of rolling bearing. Aiming at the few-shot issue and multi-label attributes of single-point faults, a novel fault diagnosis method of rolling bearing based on time–frequency signature matrix (T–FSM) feature and multi-label convolutional neural network with meta-learning (MLCML) is proposed in this paper. At the beginning, the T–FSM features sensitive to few-shot fault diagnosis of measured vibration signal are extracted. Subsequently, a designed multi-label convolutional neural network (MLCNN) with a specific architecture is employed to identify faults. Crucially, the meta-learning strategy of learning initial network parameters susceptive to task changes is incorporated to MLCNN for addressing the few-shot problem. Ultimately, the publicly available rolling bearing dataset is utilized to demonstrate the effectiveness of the proposed method. The experimental results exhibit that the trained MLCML has the capability of learning to learn few-shot fault attributes with outstanding diagnosis accuracy and generalization. More concretely, the model can adapt to new fault categories rapidly owing to that only a few samples and update steps are required to fine-tune the network.

Recommendations for Different Tasks Based on the Uniform Multimodal Joint Representation

Content curation social networks (CCSNs), such as Pinterest and Huaban, are interest driven and content centric. On CCSNs, user interests are represented by a set of boards, and a board is composed of various pins. A pin is an image with a description. All entities, such as users, boards, and categories, can be represented as a set of pins. Therefore, it is possible to implement entity representation and the corresponding recommendations on a uniform representation space from pins. Furthermore, lots of pins are re-pinned from others and the pin’s re-pin sequences are recorded on CCSNs. In this paper, a framework which can learn the multimodal joint representation of pins, including text representation, image representation, and multimodal fusion, is proposed. Image representations are extracted from a multilabel convolutional neural network. The multiple labels of pins are automatically obtained by the category distributions in the re-pin sequences, which benefits from the network architecture. Text representations are obtained with the word2vec tool. Two modalities are fused with a multimodal deep Boltzmann machine. On the basis of the pin representation, different recommendation tasks are implemented, including recommending pins or boards to users, recommending thumbnails to boards, and recommending categories to boards. Experimental results on a dataset from Huaban demonstrate that the multimodal joint representation of pins contains the information of user interests. Furthermore, the proposed multimodal joint representation outperformed unimodal representation in different recommendation tasks. Experiments were also performed to validate the effectiveness of the proposed recommendation methods.

A Deep Multi-Label Learning Framework for the Intelligent Fault Diagnosis of Machines

Deep learning has been applied in intelligent fault diagnosis of machines since it trains deep neural networks to simultaneously learn features and recognize faults. In the intelligent fault diagnosis methods based on deep learning, feature learning and fault recognition are achieved by solving a multi-class classification problem. The multi-class classification, however, has not considered the relationships of fault labels, leading to two weaknesses of these methods. One is that it cannot ensure to learn the correlated features for related faults and the other is that it cannot handle missing label problem. To overcome these weaknesses, we introduce a concept of multi-label classification into intelligent fault diagnosis and propose a deep multi-label learning framework called multi-label convolutional neural network (MLCNN). MLCNN builds the relationship between the labels, and thus it is able to learn the correlated features from mechanical vibration signals and be well trained using the samples with missing labels. A motor bearing diagnosis case and a compound fault diagnosis case are used to verify the proposed method, respectively. The results show that the relationships between features are learned by MLCNN, and the classification accuracies of MLCNN are higher than traditional methods when the missing label problem occurs.