Multi-task Convolutional Neural Network Research Articles

AI-based 3D analysis of retinal vasculature associated with retinal diseases using OCT angiography

Retinal vasculature is the only vascular system in the human body that can be observed in a non-invasive manner, with a phenotype associated with a wide range of ocular, cerebral, and cardiovascular diseases. OCT and OCT angiography (OCTA) provide powerful imaging methods to visualize the three-dimensional morphological and functional information of the retina. In this study, based on OCT and OCTA multimodal inputs, a multitask convolutional neural network model was built to realize 3D segmentation of retinal blood vessels and disease classification for different retinal diseases, overcoming the limitations of existing methods that can only perform 2D analysis of OCTA. Two hundred thirty sets of OCT and OCTA data from 109 patients, including 138,000 cross-sectional images in normal and diseased eyes (age-related macular degeneration, retinal vein occlusion, and central serous chorioretinopathy), were collected from four commercial OCT systems for model training, validation, and testing. Experimental results verified that the proposed method was able to achieve a DICE coefficient of 0.956 for 3D segmentation of blood vessels and an accuracy of 91.49% for disease classification, and further enabled us to evaluate the 3D reconstruction of retinal vessels, explore the interlayer connections of superficial and deep vasculatures, and reveal the 3D quantitative vessel characteristics in different retinal diseases.

MTU-Net: Multi-Task Convolutional Neural Network for Breast Calcification Segmentation from Mammograms

MTU-Net: Multi-Task Convolutional Neural Network for Breast Calcification Segmentation from Mammograms

Detection of macular neovascularization in eyes presenting with macular edema using optical coherence tomography angiography and a deep-learning model

PurposeTo test the diagnostic performance of an artificial intelligence algorithm for detecting and segmenting macular neovascularization (MNV) with optical coherence tomography (OCT) and OCT angiography(OCTA) in eyes with macular edema from various diagnoses. DesignProspective cross-sectional study. ParticipantsStudy participants with macular edema due to either treatment-naïve exudative age-related macular degeneration (AMD), diabetic macular edema (DME), or retinal vein occlusion (RVO). MethodsStudy participants were imaged with macular 3x3-mm and 6x6-mm spectral-domain OCTA. Eyes with exudative AMD were required to have MNV in the central 3x3-mm area. A previously developed hybrid multi-task convolutional neural network for MNV detection (aiMNV) and segmentation was applied to all images, regardless of image quality. Main Outcome MeasuresSensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of detecting MNV; and intersection over union(IoU) score and F1 score for segmentation. ResultsOf 114 eyes from 112 study participants, 56 eyes had MNV due to exudative AMD and 58 eyes with macular edema due to either DME or RVO. 3x3-mm OCTA scans with aiMNV detected MNV with 96.4% sensitivity, 98.3% specificity, 98.2% PPV, and 96.6% NPV. For segmentation, the average IoU score was 0.947 and the F1 score was 0.973. 6x6-mm scans performed well; however, sensitivity for MNV detection was lower than 3x3-mm scans due to lower scan sampling density. ConclusionThis novel aiMNV algorithm can accurately detect and segment MNV in eyes with exudative AMD from a control group of eyes that present with macular edema from either DME or RVO. Higher scan sampling density improved the aiMNV sensitivity for MNV detection.

Development of message passing-based graph convolutional networks for classifying cancer pathology reports

BackgroundApplying graph convolutional networks (GCN) to the classification of free-form natural language texts leveraged by graph-of-words features (TextGCN) was studied and confirmed to be an effective means of describing complex natural language texts. However, the text classification models based on the TextGCN possess weaknesses in terms of memory consumption and model dissemination and distribution. In this paper, we present a fast message passing network (FastMPN), implementing a GCN with message passing architecture that provides versatility and flexibility by allowing trainable node embedding and edge weights, helping the GCN model find the better solution. We applied the FastMPN model to the task of clinical information extraction from cancer pathology reports, extracting the following six properties: main site, subsite, laterality, histology, behavior, and grade.ResultsWe evaluated the clinical task performance of the FastMPN models in terms of micro- and macro-averaged F1 scores. A comparison was performed with the multi-task convolutional neural network (MT-CNN) model. Results show that the FastMPN model is equivalent to or better than the MT-CNN.ConclusionsOur implementation revealed that our FastMPN model, which is based on the PyTorch platform, can train a large corpus (667,290 training samples) with 202,373 unique words in less than 3 minutes per epoch using one NVIDIA V100 hardware accelerator. Our experiments demonstrated that using this implementation, the clinical task performance scores of information extraction related to tumors from cancer pathology reports were highly competitive.

Monocular Pose and Shape Reconstruction of Vehicles in UAV imagery using a Multi-task CNN

Estimating the pose and shape of vehicles from aerial images is an important, yet challenging task. While there are many existing approaches that use stereo images from street-level perspectives to reconstruct objects in 3D, the majority of aerial configurations used for purposes like traffic surveillance are limited to monocular images. Addressing this challenge, a Convolutional Neural Network-based method is presented in this paper, which jointly performs detection, pose, type and 3D shape estimation for vehicles observed in monocular UAV imagery. For this purpose, a robust 3D object model is used following the concept of an Active Shape Model. In addition, different variants of loss functions for learning 3D shape estimation are presented, focusing on the height component, which is particularly challenging to estimate from monocular near-nadir images. We also introduce a UAV-based dataset to evaluate our model in addition to an augmented version of the publicly available Hessigheim benchmark dataset. Our method yields promising results in pose and shape estimation: utilising images with a ground sampling distance (GSD) of 3 cm, it achieves median errors of up to 4 cm in position and 3° in orientation. Additionally, it achieves root mean square (RMS) errors of \\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\pm 6$$\\end{document} cm in planimetry and \\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\pm 18$$\\end{document} cm in height for keypoints defining the car shape.

A Multi-Task Convolutional Neural Network Relative Radiometric Calibration Based on Temporal Information

Due to the continuous degradation of onboard satellite instruments over time, satellite images undergo degradation, necessitating calibration for tasks reliant on satellite data. The previous relative radiometric calibration methods are mainly categorized into traditional methods and deep learning methods. The traditional methods involve complex computations for each calibration, while deep-learning-based approaches tend to oversimplify the calibration process, utilizing generic computer vision models without tailored structures for calibration tasks. In this paper, we address the unique challenges of calibration by introducing a novel approach: a multi-task convolutional neural network calibration model leveraging temporal information. This pioneering method is the first to integrate temporal dynamics into the architecture of neural network calibration models. Extensive experiments conducted on the FY3A/B/C VIRR datasets showcase the superior performance of our approach compared to the existing state-of-the-art traditional and deep learning methods. Furthermore, tests with various backbones confirm the broad applicability of our framework across different convolutional neural networks.

A Survey an Optimized Dense CNN Model for Recognizing Deepfake Images

Deepfake detection relies on a deep learning model. Deepfake content is generated using artificial intelligence and machine learning techniques to swap one person’s face with another’s in images. These altered images are certain to have a significant impact on society. Deepfakes utilize advanced technologies such as machine learning (ML) and deep learning (DL) to develop automated techniques for generating deceptive content. The model is trained using the DFDC (Deepfake Detection Challenge) dataset, which includes 1, 00,000 videos comprising both real and fake content. This survey examines recent progress in deep learning techniques for detecting deepfake images, highlighting their growing significance in today’s digital landscape. The paper outlines the issues arising from deepfakes, explores the techniques for generating deceptive content, and analyzes various deep learning models employed in deepfake detection, such as convolutional neural networks (CNN’s), MTCNN (Multi-task CNN), and Facial Landmark. It emphasizes the ongoing battle between the progression of deepfakes and the evolution of detection methods, underscoring the need for advanced and flexible neural network architectures to effectively curb the dissemination of deceptive information.

Deep Learning with a Multi-Task Convolutional Neural Network to Generate a National-Scale 3D Soil Data Product: The Particle Size Distribution of the German Agricultural Soil Landscape

Deep Learning with a Multi-Task Convolutional Neural Network to Generate a National-Scale 3D Soil Data Product: The Particle Size Distribution of the German Agricultural Soil Landscape

Bayesian multivariate receptor model and convolutional neural network to identify quantitative sources and spatial distributions of potentially toxic elements in soils: A case study in Qingzhou City, China

Determining sources and spatial distributions of potentially toxic elements (PTEs) is a crucial issue of soil pollution survey. However, uncertainty estimation for source contributions remains lack, and accurate spatial prediction is still challenging. Robust Bayesian multivariate receptor model (RBMRM) was applied to the soil dataset of Qingzhou City (8 PTEs in 429 samples), to calculate source contributions with uncertainties. Multi-task convolutional neural network (MTCNN) was proposed to predict spatial distributions of soil PTEs. RBMRM afforded three sources, consistent with US-EPA positive matrix factorization. Natural source dominated As, Cr, Cu, and Ni contents (78.5 %∼86.1 %), and contributed 37.1 %, 61.0 %, and 65.9 % of Cd, Pb, and Zn, exhibiting low uncertainties with uncertainty index (UI) < 26.7 %. Industrial, traffic, and agricultural sources had significant influences on Cd, Pb, and Zn (30.2 %∼61.9 %), with UI < 39.3 %. Hg originated dominantly from atmosphere deposition (99.1 %), with relatively high uncertainties (UI=87.7 %). MTCNN acquired satisfactory accuracies, with R2 of 0.357–0.896 and nRMSE of 0.092–0.366. Spatial distributions of As, Cd, Cr, Cu, Ni, Pb, and Zn were influenced by parent materials. Cd, Hg, Pb, and Zn showed significant hotspot in urban area. This work conducted a new approach exploration, and practical implications for soil pollution regulation were proposed.

Chemical composition prediction in goji (Lycium barbarum) using hyperspectral imaging and multi-task 1DCNN with attention mechanism

The bioactive components of goji berries (Lycium barbarum) are crucial determinants of their nutritional and commercial value. In this study, we combined hyperspectral imaging technology (HSI) with a one-dimensional convolutional neural network (1DCNN) to predict the content of chemical compositions in goji. To enhance the model's ability to focus on relevant information, we introduced the channel attention module (CAM), spectral attention module (SAM) and fused them together, which can focus on output features of the convolution kernels differently and adaptively emphasize more effective spectral bands, respectively. Moreover, considering the limitations of traditional single-task prediction methods and the inherent correlations among different constituents, we employed a multi-task CNN for the simultaneous prediction of various goji constituents. The results indicate that the attention-enhanced 1DCNN model outperforms both the partial least squares regression (PLSR) model and the vanilla 1DCNN. With multi-task learning, the model achieves optimal performance, achieving an average R2 of 0.9435 for the prediction of the three components. Our research develops an efficient and accurate method for predicting the constituents of goji berries, providing a valuable, convenient, and effective tool for assessing and detecting the quality of fruit products in the food industry.

A Multi-task CNN Model for Automated Prediction of Isocitrate Dehydrogenase Mutation Status and Grade of Gliomas

A Multi-task CNN Model for Automated Prediction of Isocitrate Dehydrogenase Mutation Status and Grade of Gliomas

Signal separation of simultaneous dual-tracer PET imaging based on global spatial information and channel attention

BackgroundSimultaneous dual-tracer positron emission tomography (PET) imaging efficiently provides more complete information for disease diagnosis. The signal separation has long been a challenge of dual-tracer PET imaging. To predict the single-tracer images, we proposed a separation network based on global spatial information and channel attention, and connected it to FBP-Net to form the FBPnet-Sep model.ResultsExperiments using simulated dynamic PET data were conducted to: (1) compare the proposed FBPnet-Sep model to Sep-FBPnet model and currently existing Multi-task CNN, (2) verify the effectiveness of modules incorporated in FBPnet-Sep model, (3) investigate the generalization of FBPnet-Sep model to low-dose data, and (4) investigate the application of FBPnet-Sep model to multiple tracer combinations with decay corrections. Compared to the Sep-FBPnet model and Multi-task CNN, the FBPnet-Sep model reconstructed single-tracer images with higher structural similarity, peak signal-to-noise ratio and lower mean squared error, and reconstructed time-activity curves with lower bias and variation in most regions. Excluding the Inception or channel attention module resulted in degraded image qualities. The FBPnet-Sep model showed acceptable performance when applied to low-dose data. Additionally, it could deal with multiple tracer combinations. The qualities of predicted images, as well as the accuracy of derived time-activity curves and macro-parameters were slightly improved by incorporating a decay correction module.ConclusionsThe proposed FBPnet-Sep model was considered a potential method for the reconstruction and signal separation of simultaneous dual-tracer PET imaging.

Computer-based facial recognition as an assisting diagnostic tool to identify children with Noonan syndrome

BackgroundNoonan syndrome (NS) is a rare genetic disease, and patients who suffer from it exhibit a facial morphology that is characterized by a high forehead, hypertelorism, ptosis, inner epicanthal folds, down-slanting palpebral fissures, a highly arched palate, a round nasal tip, and posteriorly rotated ears. Facial analysis technology has recently been applied to identify many genetic syndromes (GSs). However, few studies have investigated the identification of NS based on the facial features of the subjects.ObjectivesThis study develops advanced models to enhance the accuracy of diagnosis of NS.MethodsA total of 1,892 people were enrolled in this study, including 233 patients with NS, 863 patients with other GSs, and 796 healthy children. We took one to 10 frontal photos of each subject to build a dataset, and then applied the multi-task convolutional neural network (MTCNN) for data pre-processing to generate standardized outputs with five crucial facial landmarks. The ImageNet dataset was used to pre-train the network so that it could capture generalizable features and minimize data wastage. We subsequently constructed seven models for facial identification based on the VGG16, VGG19, VGG16-BN, VGG19-BN, ResNet50, MobileNet-V2, and squeeze-and-excitation network (SENet) architectures. The identification performance of seven models was evaluated and compared with that of six physicians.ResultsAll models exhibited a high accuracy, precision, and specificity in recognizing NS patients. The VGG19-BN model delivered the best overall performance, with an accuracy of 93.76%, precision of 91.40%, specificity of 98.73%, and F1 score of 78.34%. The VGG16-BN model achieved the highest AUC value of 0.9787, while all models based on VGG architectures were superior to the others on the whole. The highest scores of six physicians in terms of accuracy, precision, specificity, and the F1 score were 74.00%, 75.00%, 88.33%, and 61.76%, respectively. The performance of each model of facial recognition was superior to that of the best physician on all metrics.ConclusionModels of computer-assisted facial recognition can improve the rate of diagnosis of NS. The models based on VGG19-BN and VGG16-BN can play an important role in diagnosing NS in clinical practice.

Application of Convolutional Networks for Localization and Prediction of Scalar Parameters of Fractured Geological Inclusion

Seismic inversion is an important part of the modern geological exploration process. Novel applications of deep learning are capable of handling heterogeneous media, but require too much data for training. In this paper, we focus on the prediction of fracture inclusion location and its parameters in rock media and approach the problem in the multi-task manner. For this, several multi-task convolutional neural network (CNN) architectures are proposed and compared. The direct seismic problem is considered in the heterogeneous fractured geological model based on the well-known Marmousi2 model in a two-dimensional case. The model of the deformable solid medium containing slip planes with nonlinear slip conditions on them and explicit–implicit numerical method is applied to obtain the synthetic seismic dataset for CNN training and validation.

Simultaneous determination of pigments of spinach (Spinacia oleracea L.) leaf for quality inspection using hyperspectral imaging and multi-task deep learning regression approaches

Rapid and accurate determination of pigment content is important for quality inspection of spinach leaves during storage. This study aimed to use hyperspectral imaging at two spectral ranges (visible/near-infrared, VNIR: 400–1000 nm; NIR: 900–1700 nm) to simultaneously determine the pigment (chlorophyll a, chlorophyll b, total chlorophyll, and carotenoids) content in spinach stored at different durations and conditions (unpackaged and packaged). Partial least squares (PLS), back propagation neural network (BPNN) and convolutional neural network (CNN) were used to establish single-task and multi-task regression models. Single-task CNN (STCNN) models and multi-task CNN (MTCNN) models obtained better performances than the other models. The models using VNIR spectra were superior to those using NIR spectra. The overall results indicated that hyperspectral imaging with multi-task learning could predict the quality attributes of spinach simultaneously for spinach quality inspection under various storage conditions. This research will guide food quality inspection by simultaneously inspecting multiple quality attributes.

Self-supervised motor imagery EEG recognition model based on 1-D MTCNN-LSTM network

Objective. Aiming for the research on the brain–computer interface (BCI), it is crucial to design a MI-EEG recognition model, possessing a high classification accuracy and strong generalization ability, and not relying on a large number of labeled training samples. Approach. In this paper, we propose a self-supervised MI-EEG recognition method based on self-supervised learning with one-dimensional multi-task convolutional neural networks and long short-term memory (1-D MTCNN-LSTM). The model is divided into two stages: signal transform identification stage and pattern recognition stage. In the signal transform recognition phase, the signal transform dataset is recognized by the upstream 1-D MTCNN-LSTM network model. Subsequently, the backbone network from the signal transform identification phase is transferred to the pattern recognition phase. Then, it is fine-tuned using a trace amount of labeled data to finally obtain the motion recognition model. Main results. The upstream stage of this study achieves more than 95% recognition accuracy for EEG signal transforms, up to 100%. For MI-EEG pattern recognition, the model obtained recognition accuracies of 82.04% and 87.14% with F1 scores of 0.7856 and 0.839 on the datasets of BCIC-IV-2b and BCIC-IV-2a. Significance. The improved accuracy proves the superiority of the proposed method. It is prospected to be a method for accurate classification of MI-EEG in the BCI system.

Joint learning strategy of multi-scale multi-task convolutional neural network for aero-engine prognosis

Remaining useful life (RUL) prediction and health status (HS) assessment are two key tasks in aero-engine prognostics and health management (PHM) system. However, existing deep learning-based prognostic models perform RUL prediction and HS assessment tasks separately, without considering the correlation between these two tasks. Secondly, traditional deep learning can only extract single-scale features, which limits the ability to extract complex degradation features from high-dimensional condition monitoring data. Therefore, this work proposes a multi-scale and multi-task convolutional neural network for joint learning of aero-engine RUL prediction and HS assessment. Firstly, multi-sensor data with multiple cycles are converted into image samples to integrate more condition monitoring information that is beneficial to prognosis. Then, the multi-scale feature fusion block is designed as the shared network for multi-task, utilizing convolutional layers with filters of different sizes to enhance the ability to extract complex degradation features from high-dimensional condition monitoring data. And a multi-layer concatenation block is constructed to integrate multi-scale features at different levels to fully utilize the important information at different levels. On this basis, a multi-task joint learning block is constructed and a joint loss function is developed for joint learning of RUL prediction and HS assessment. Finally, experiments on two engine degradation datasets, CMAPSS and N-CMAPSS, demonstrate that the proposed network has excellent RUL prediction and HS assessment performance, and outperforms other state-of-the-art methods.

Preoperative CECT-Based Multitask Model Predicts Peritoneal Recurrence and Disease-Free Survival in Advanced Ovarian Cancer: A Multicenter Study

ObjectivePeritoneal recurrence is the predominant pattern of recurrence in advanced ovarian cancer (AOC) and portends a dismal prognosis. Accurate prediction of peritoneal recurrence and disease-free survival (DFS) is crucial to identify patients who might benefit from intensive treatment. We aimed to develop a predictive model for peritoneal recurrence and prognosis in AOC. MethodsIn this retrospective multi-institution study of 515 patients, an end-to-end multi-task convolutional neural network (MCNN) comprising a segmentation convolutional neural network (CNN) and a classification CNN was developed and tested using preoperative CT images, and MCNN-score was generated to indicate the peritoneal recurrence and DFS status in patients with AOC. We evaluated the accuracy of the model for automatic segmentation and predict prognosis. ResultsThe MCNN achieved promising segmentation performances with a mean Dice coefficient of 84.3% (range: 78.8%-87.0%). The MCNN was able to predict peritoneal recurrence in the training (AUC 0.87; 95% CI 0.82–0.90), internal test (0.88; 0.85–0.92), and external test set (0.82; 0.78–0.86). Similarly, MCNN demonstrated consistently high accuracy in predicting recurrence, with an AUC of 0.85; 95% CI 0.82–0.88, 0.83; 95% CI 0.80–0.86, and 0.85; 95% CI 0.83–0.88. For patients with a high MCNN-score of recurrence, it was associated with poorer DFS with P<0.0001 and hazard ratios of 0.1964 (95% CI: 0.1439-0.2680), 0.3249 (95% CI: 0.1896-0.5565), and 0.3458 (95% CI: 0.2582-0.4632). ConclusionsThe MCNN approach demonstrated high performance in predicting peritoneal recurrence and DFS in patients with AOC.

A deep learning framework for 3D vegetation extraction in complex urban environments

Accurate extraction of three-dimensional (3D) vegetation is essential for monitoring urban ecological environments and carbon sinks. Two-dimensional vegetation data in cities has been widely researched. However, large-scale urban vegetation height inventories are lacking. This study proposes a novel framework for 3D extraction of urban vegetation, which can be widely applied based on remote sensing approaches. A multi-task convolutional neural network is established to extract the urban vegetation cover and estimate the vegetation height at the pixel level. The results indicate that this method can derive the complete urban vegetation cover and height from stereo satellite data. Compared with the traditional stereo-photogrammetry method, this method enables rapid inference of vegetation height in urban areas with a root mean square error (RMSE) of 3.16 m. This model is capable of accurately separating vegetation in complex urban environments and performs well despite shadow effects. Furthermore, in this study, the first vegetation height map with 1-m spatial resolution has been produced, covering six urban districts in Beijing (approximately 1,378 km2). It only takes 2–3 min to process the imagery of the whole study area. The high-resolution map can display more urban vegetation details over the existing 10 m/30 m resolution vegetation height maps. Furthermore, the established framework and benchmark for urban vegetation 3D information offer unique insights and provide a basis for further research.

Automatic diagnosis of depression based on attention mechanism and feature pyramid model.

Currently, most diagnoses of depression are evaluated by medical professionals, with the results of these evaluations influenced by the subjective judgment of physicians. Physiological studies have shown that depressed patients display facial movements, head posture, and gaze direction disorders. To accurately diagnose the degree of depression of patients, this paper proposes a comprehensive framework, Cross-Channel Attentional Depression Detection Network, which can automatically diagnose the degree of depression of patients by inputting information from the facial images of depressed patients. Specifically, the comprehensive framework is composed of three main modules: (1) Face key point detection and cropping for video images based on Multi-Task Convolutional Neural Network. (2) The improved Feature Pyramid Networks model can fuse shallow features and deep features in video images and reduce the loss of miniscule features. (3) A proposed Cross-Channel Attention Convolutional Neural Network can enhance the interaction between tensor channel layers. Compared to other methods for automatic depression identification, a superior method was obtained by conducting extensive experiments on the depression dataset AVEC 2014, where the Root Mean Square Error and the Mean Absolute Error were 8.65 and 6.66, respectively.