Multi-task Convolutional Neural Network Research Articles

Joint learning strategy of multi-scale multi-task convolutional neural network for aero-engine prognosis

Remaining useful life (RUL) prediction and health status (HS) assessment are two key tasks in aero-engine prognostics and health management (PHM) system. However, existing deep learning-based prognostic models perform RUL prediction and HS assessment tasks separately, without considering the correlation between these two tasks. Secondly, traditional deep learning can only extract single-scale features, which limits the ability to extract complex degradation features from high-dimensional condition monitoring data. Therefore, this work proposes a multi-scale and multi-task convolutional neural network for joint learning of aero-engine RUL prediction and HS assessment. Firstly, multi-sensor data with multiple cycles are converted into image samples to integrate more condition monitoring information that is beneficial to prognosis. Then, the multi-scale feature fusion block is designed as the shared network for multi-task, utilizing convolutional layers with filters of different sizes to enhance the ability to extract complex degradation features from high-dimensional condition monitoring data. And a multi-layer concatenation block is constructed to integrate multi-scale features at different levels to fully utilize the important information at different levels. On this basis, a multi-task joint learning block is constructed and a joint loss function is developed for joint learning of RUL prediction and HS assessment. Finally, experiments on two engine degradation datasets, CMAPSS and N-CMAPSS, demonstrate that the proposed network has excellent RUL prediction and HS assessment performance, and outperforms other state-of-the-art methods.

Preoperative CECT-Based Multitask Model Predicts Peritoneal Recurrence and Disease-Free Survival in Advanced Ovarian Cancer: A Multicenter Study

ObjectivePeritoneal recurrence is the predominant pattern of recurrence in advanced ovarian cancer (AOC) and portends a dismal prognosis. Accurate prediction of peritoneal recurrence and disease-free survival (DFS) is crucial to identify patients who might benefit from intensive treatment. We aimed to develop a predictive model for peritoneal recurrence and prognosis in AOC. MethodsIn this retrospective multi-institution study of 515 patients, an end-to-end multi-task convolutional neural network (MCNN) comprising a segmentation convolutional neural network (CNN) and a classification CNN was developed and tested using preoperative CT images, and MCNN-score was generated to indicate the peritoneal recurrence and DFS status in patients with AOC. We evaluated the accuracy of the model for automatic segmentation and predict prognosis. ResultsThe MCNN achieved promising segmentation performances with a mean Dice coefficient of 84.3% (range: 78.8%-87.0%). The MCNN was able to predict peritoneal recurrence in the training (AUC 0.87; 95% CI 0.82–0.90), internal test (0.88; 0.85–0.92), and external test set (0.82; 0.78–0.86). Similarly, MCNN demonstrated consistently high accuracy in predicting recurrence, with an AUC of 0.85; 95% CI 0.82–0.88, 0.83; 95% CI 0.80–0.86, and 0.85; 95% CI 0.83–0.88. For patients with a high MCNN-score of recurrence, it was associated with poorer DFS with P<0.0001 and hazard ratios of 0.1964 (95% CI: 0.1439-0.2680), 0.3249 (95% CI: 0.1896-0.5565), and 0.3458 (95% CI: 0.2582-0.4632). ConclusionsThe MCNN approach demonstrated high performance in predicting peritoneal recurrence and DFS in patients with AOC.

A deep learning framework for 3D vegetation extraction in complex urban environments

Accurate extraction of three-dimensional (3D) vegetation is essential for monitoring urban ecological environments and carbon sinks. Two-dimensional vegetation data in cities has been widely researched. However, large-scale urban vegetation height inventories are lacking. This study proposes a novel framework for 3D extraction of urban vegetation, which can be widely applied based on remote sensing approaches. A multi-task convolutional neural network is established to extract the urban vegetation cover and estimate the vegetation height at the pixel level. The results indicate that this method can derive the complete urban vegetation cover and height from stereo satellite data. Compared with the traditional stereo-photogrammetry method, this method enables rapid inference of vegetation height in urban areas with a root mean square error (RMSE) of 3.16 m. This model is capable of accurately separating vegetation in complex urban environments and performs well despite shadow effects. Furthermore, in this study, the first vegetation height map with 1-m spatial resolution has been produced, covering six urban districts in Beijing (approximately 1,378 km2). It only takes 2–3 min to process the imagery of the whole study area. The high-resolution map can display more urban vegetation details over the existing 10 m/30 m resolution vegetation height maps. Furthermore, the established framework and benchmark for urban vegetation 3D information offer unique insights and provide a basis for further research.

Automatic diagnosis of depression based on attention mechanism and feature pyramid model.

Currently, most diagnoses of depression are evaluated by medical professionals, with the results of these evaluations influenced by the subjective judgment of physicians. Physiological studies have shown that depressed patients display facial movements, head posture, and gaze direction disorders. To accurately diagnose the degree of depression of patients, this paper proposes a comprehensive framework, Cross-Channel Attentional Depression Detection Network, which can automatically diagnose the degree of depression of patients by inputting information from the facial images of depressed patients. Specifically, the comprehensive framework is composed of three main modules: (1) Face key point detection and cropping for video images based on Multi-Task Convolutional Neural Network. (2) The improved Feature Pyramid Networks model can fuse shallow features and deep features in video images and reduce the loss of miniscule features. (3) A proposed Cross-Channel Attention Convolutional Neural Network can enhance the interaction between tensor channel layers. Compared to other methods for automatic depression identification, a superior method was obtained by conducting extensive experiments on the depression dataset AVEC 2014, where the Root Mean Square Error and the Mean Absolute Error were 8.65 and 6.66, respectively.

An efficient multi-task learning CNN for driver attention monitoring

Driver Monitoring System (DMS), usually equipped with a camera, is an emerging vehicle safety system that can monitor driver attentiveness and trigger timely alarms when signs of inattention are detected. Since a single indicator (e.g., eye blink rate) is insufficient and unreliable to analyze driver attentiveness, almost all existing solutions train several independent models to identify driver facial states, such as face landmark, head pose, yawning, eye state, etc. However, apart from neglecting the inherent correlations between these related tasks, multiple models also raise challenges for vehicle safety-critical systems (e.g., hardware resources, software compatibility, and real-time response). In this paper, we propose a multi-task learning CNN framework (DANet) to unify the relevant tasks into one model and simultaneously output various driver facial states. By sharing the common features and parameters of highly related tasks, DANet avoids repetitive computations and mitigates single task overfitting. More importantly, the model provides a comprehensive overview of facial states while maintaining low complexity. We also propose two novel designs: (1) Dual-loss Block, which decomposes the pose estimation task into pose classification and coarse-to-fine regression; (2) Head Pose Penalization, which constrains the network to predict gaze direction based on predicted head pose. Our method achieves compelling results in both speed and accuracy on a vehicle computing platform, marking a momentous step in this field.

Recognition of mango and location of picking point on stem based on a multi-task CNN model named YOLOMS

Recognition of mango and location of picking point on stem based on a multi-task CNN model named YOLOMS

Age and sex estimation in cephalometric radiographs based on multitask convolutional neural networks

ObjectivesAge and sex characteristics are evident in cephalometric radiographs (CRs), yet their accurate estimation remains challenging due to the complexity of these images. This study aimed to harness deep learning to automate age and gender estimation from CRs, potentially simplifying their interpretation. Study DesignWe compared the performance of 4 deep learning models (SVM, R-net, VGG16-SingleTask, and our proposed VGG16-MultiTask) in estimating age and sex from the testing dataset, utilizing a VGG16-based multitask deep learning model on 4,557 CRs. Gradient-weighted class activation mapping (Grad-CAM) was incorporated to identify sex. Performance was assessed using mean absolute error (MAE), specificity, sensitivity, F1 score, and the area under the curve (AUC) in receiver operating characteristic analysis. ResultsThe VGG16-MultiTask model outperformed the others, with the lowest MAE (0.864±1.602) and highest sensitivity (0.85), specificity (0.88), F1 score (0.863), and AUC (0.93), demonstrating superior precision and robust performance. ConclusionsThe VGG multitask model demonstrates significant potential in enhancing age and sex estimation from cephalometric analysis, underscoring the role of AI in improving biomedical interpretations.

Comprehensive visual information acquisition for tomato picking robot based on multitask convolutional neural network

The tomato picking robot's vision system faces two difficult tasks: precise tomato pose acquisition and stem location. Tomato pose and stem location can help determine the end effector pose and achieve collision-free picking. To realise efficient crop picking, the tasks of target location, pose detection, and obstacle semantic segmentation should be completed in one model to obtain comprehensive visual information. Therefore, the multitask convolutional neural network YOLO-MCNN is proposed, a new method to complete the above tasks in one model. By fusing multi-scale features and determining the optimal locations for the semantic segmentation branch, four strategies are proposed for enhancing the segmentation ability. The experiment results show that fusing the semantic segmentation branch with the second layer of shallow feature maps and placing the branch after the 17th layer can result in the best segmentation performance. Fusing shallow feature maps improves small target detection while merging multi-scale feature maps enhances semantic segmentation performance. Moreover, ablation experiments are conducted to understand the influence between multitask convolutional and single task networks. It proves that running multiple tasks on the same backbone network does not affect their performance. The YOLO-MCNN's target detection performance F1 is 87.8%, the semantic segmentation performance mIoU is 74.8%, the keypoint detection performance dlmk is 6.95 pixels, the network size is 15.2 MB, and the inference speed is 19.9ms. Compared with other target detection and semantic segmentation networks, it shows that the comprehensive performance of the YOLO-MCNN is the best. The method provides theoretical foundation for constructing multitask convolutional neural networks.

Multi-Task Convolutional Neural Network for Image Aesthetic Assessment

Multi-Task Convolutional Neural Network for Image Aesthetic Assessment

Simultaneous Hip Implant Segmentation and Gruen Landmarks Detection.

The assessment of implant status and complications of Total Hip Replacement (THR) relies mainly on the clinical evaluation of the X-ray images to analyse the implant and the surrounding rigid structures. Current clinical practise depends on the manual identification of important landmarks to define the implant boundary and to analyse many features in arthroplasty X-ray images, which is time-consuming and could be prone to human error. Semantic segmentation based on the Convolutional Neural Network (CNN) has demonstrated successful results in many medical segmentation tasks. However, these networks cannot define explicit properties that lead to inaccurate segmentation, especially with the limited size of image datasets. Our work integrates clinical knowledge with CNN to segment the implant and detect important features simultaneously. This is instrumental in the diagnosis of complications of arthroplasty, particularly for loose implant and implant-closed bone fractures, where the location of the fracture in relation to the implant must be accurately determined. In this work, we define the points of interest using Gruen zones that represent the interface of the implant with the surrounding bone to build a Statistical Shape Model (SSM). We propose a multitask CNN that combines regression of pose and shape parameters constructed from the SSM and semantic segmentation of the implant. This integrated approach has improved the estimation of implant shape, from 74% to 80% dice score, making segmentation realistic and allowing automatic detection of Gruen zones. To train and evaluate our method, we generated a dataset of annotated hip arthroplasty X-ray images that will be made available.

Intelligent education management system design for universities based on MTCNN face recognition algorithm

Abstract This paper first constructs an intelligent education management system for universities according to their needs and explains its logical and technical architecture. Secondly, a multi-task convolutional neural network with MTCNN is used to detect faces, and different loss functions are selected to ensure recognition accuracy. Finally, the effectiveness of the application of intelligent education management systems in universities is verified through performance tests. The results show that the time consumed for face recognition detection using MTCNN is 524ms, and the response time of each function of the system does not exceed 500ms when the number of concurrent users is less than 500, which indicates that the intelligent education management system based on MTCNN face recognition algorithm has a good response speed and meets the requirements of intelligent education management in colleges and universities.

Deep Concrete Flow: Deep learning based characterisation of fresh concrete properties from open-channel flow using spatio-temporal flow fields

The global production of concrete, being one of the most commonly utilised materials in the world, is associated with significant CO2-emissions and a substantial depletion of mineral resources. In order to improve sustainability, concretes thus are increasingly produced using recipes containing a large variety of different raw materials, including e.g. CO2 reduced cements or recycled materials and industrial wastes. However, these actions result in heightened susceptibility of the concrete to variations in raw material characteristics, consequently diminishing the concrete’s resilience and decreasing the concrete’s robustness. Against this background, the quality control of fresh concrete before casting becomes of significant importance. However, current quality control is mainly conducted based on analogous, empirical, and often subjective test approaches. In this paper, a novel method is introduced for automatically and comprehensively evaluating the quality of fresh concrete at construction sites. In particular, the paper presents an end-to-end trainable framework for the image-based characterisation of fresh concrete properties. More specifically, a camera-based setup is proposed, in which image sequences of the discharge process of a mixing truck are acquired, on the basis of which the fresh concrete properties such as the consistency and rheology are determined. In this context, this paper introduces the concept of Spatio-Temporal Flow Fields, which provide a compact representation of the concrete’s flow behaviour and serve as input to a multi-task convolutional neural network (CNN) predicting the target parameters describing the fresh concrete’s properties. A thorough examination of the performance of the suggested method is carried out using highly challenging real-world data, showcasing extremely compelling outcomes with average prediction errors for the concrete properties of only about 6%.

A deep learning-based face recognition attendance system

Deep learning-based face recognition system have produced high accuracy and better performance when compared to other methods of face recognition like the eigen faces. Modern face recognition systems consist of different phases such as face detection, face alignment, feature extraction, face representation and face recognition. This paper proposes a deep learning approach in developing a face recognition-based class attendance system. The Multitask Convolutional Neural Network (MTCNN) is used for the face detection and alignment phase and a lightweight hybrid high performance Deepface Python framework based on the ‘Deepface’ Deep Convolutional Neural Network is employed for the feature extraction, face representation and face recognition phases with FaceNet-512 pretrained model. Because Convolutional Neural Networks (CNNs) perform better with larger datasets, image augmentation will be used on the original photos to enlarge the tiny dataset. The attendance record is stored in a MySQL database and accessed by an Application Programming Interface (API) developed using Hypertext Pre-processor (PHP) CodeIgniter framework. Cosine Similarity is used as the similarity metrics to compare the facial embeddings. A sliding camera system is deployed to aids the full coverage of the class participants irrespective of the size of the class. The test result show that all class participants were correctly identified and captured in the class attendance register generated.

Automatic ultrasound diagnosis of thyroid nodules: a combination of deep learning and KWAK TI-RADS

Objective. There has been a considerable amount of computer-aided diagnosis (CAD) methods highlighted in the field of ultrasonic examination (USE) of thyroid nodules. However, few researches focused on the automatic risk classification, which was the basis for determining whether fine needle aspiration (FNA) was needed. The aim of this work was to implement automatic risk level assessment of thyroid nodules. Approach. Firstly, 1862 cases of thyroid nodules with the results of USE and FNA were collected as the dataset. Then, an improved U-Net++ model was utilized for segmenting thyroid nodules in ultrasound images automatically. Finally, the segmentation result was imported into a multi-task convolutional neural network (MT-CNN), the design of which was based on the clinical guideline called KWAK TI-RADS. Apart from the category of benign and malignant, the MT-CNN also exported the classification result of four malignant features, solid component (SC), hypoechogenicity or marked hypoechogenicity (HMH), microlobulated or irregular margin (MIM), microcalcification (MC), which were used for counting the risk level in KWAK TI-RADS. Main results. The performance of the improved U-Net++ was evaluated on our test set, including 302 cases. The Dice coefficient and intersection over union reached 0.899, 0.816, respectively. The classification accuracy rates of SC, HMH, MIM, MC, were 94.5%, 92.8%, 86.1%, 88.9%, while the false positive (FP) rate was 6.0%, 5.6%, 10.6%, 12.9% respectively. As for the category of benign and malignant, the precision and recall rates were 93.7% and 94.4%. Significance. The proposed CAD method showed favourable performance in the diagnosis of thyroid nodules. Compared with other methods, it could provide reports closer to clinical practice for doctors.

Efficient Multi-Task CNN for Face and Facial Expression Recognition Using Residual and Dense Architectures for Application in Monitoring Online Learning

Efficient Multi-Task CNN for Face and Facial Expression Recognition Using Residual and Dense Architectures for Application in Monitoring Online Learning

Eye detection and coarse localization of pupil for video-based eye tracking systems

A video-based eye tracking system generally captures NIR images, each of which contains one or two eyes of a subject. The subject’s point of gaze is then determined using 3D eye model and pupil centre corneal reflection technique. Eye detection and pupil localization play significant roles in video-based eye tracking systems. However, face rotation, wearing glasses, eye-shape variation and illumination variation make it difficult to detect an eye and localize a pupil accurately in the images captured by video-based eye tracking systems. In this paper, we proposed an eye detector that adopts a coarse-to-fine strategy. The eye detector consists of three classifiers: an ATLBP-THACs feature-based cascade classifier, a branch CNN and a multi-task CNN. We also proposed a method for coarse pupil localization. Coarse localization is an important step for pupil localization since it can provide initial pupil coordinates for a fine pupil localization method. Given a downscaled eye image, a shallow CNN is used to estimate the location of seven landmarks. On this basis, pupil center and radius are estimated. A method for small dim target enhancement is used to increase the contrast between pupil and background. The main goal of pupil enhancement is to make it easier to binarize an eye image. At last, component filtering is made by utilizing the estimated pupil center and radius. We collected a dataset named neepuEYE dataset that consists of 5500 NIR eye images from 109 people. The images can be used to generate augmented samples for training an eye detector since they contain eyes with different shape, orientation and pupil localization. Experimental results show that our eye detector is a fast and robust detector. Furthermore, our method for coarse pupil localization can obtain not only high detection rate but also high localization speed.

Adaptive Neural-Network-Based Unscented Kalman Filter for Robust Pose Tracking of Noncooperative Spacecraft

This paper presents a neural-network-based unscented Kalman filter (UKF) to estimate and track the pose (i.e., position and orientation) of a known, noncooperative, tumbling target spacecraft in a close-proximity rendezvous scenario. The UKF estimates the target’s orbit and attitude relative to the servicer based on the pose information provided by a multitask convolutional neural network (CNN) from incoming monocular images of the target. To enable reliable tracking, the process noise covariance matrix of the UKF is tuned online using adaptive state noise compensation, which leverages a newly developed closed-form process noise model for relative attitude dynamics. This paper also introduces the Satellite Hardware-In-the-loop Rendezvous Trajectories (SHIRT) dataset to enable comprehensive analyses of the performance and robustness of the proposed pipeline. SHIRT comprises the labeled images of two representative rendezvous trajectories in a low Earth orbit created using both a graphics renderer and a robotic testbed. Specifically, the CNN is solely trained on synthetic data, whereas functionality and performance of the complete navigation pipeline are evaluated on real images from the robotic testbed. The proposed UKF is evaluated on SHIRT and is shown to have subdecimeter-level position and degree-level orientation errors at steady state.

Attendance system based on facial recognition using multi-task convolutional neural network

Some businesses today do not keep track of their employees' attendance. Some of them are still using outdated techniques such as log-based manual inspections. These outdated methods of data collection require excessive paperwork, take time due to laborious recording procedures, and result in inaccurate data collection due to human error and inadequate assessment. The goal of this study is to create an attendance system that can make it simple and accurate for businesses to track their employees' attendance. The aim of this project is to create a system that will make it simple and automatic to record attendance using facial recognition. The result shows that the system is operational and possess a accurate and highly benefits the students and the faculty members.

An efficient multi-task convolutional neural network for dairy farm object detection and segmentation

Real-time and accurate detection of multiple types of targets and obstacles in dairy barns is a necessary function for autonomous pushing robots. To improve the efficiency of target recognition and to reduce the path extraction error of the pushing robot, on the basis of the high accuracy perception of every pixel collected with an embedded AI computer, a multi-task learning based dairy barn multi-type target recognition model Ghost CBAM Segmentation-Multi-task (GCS-MUL) was proposed, which could recognize dairy cows, obstacles and road targets in real-time and efficiently. Firstly, in order to enhance the ability to extract key features from the targets, the proposed model intergrades the Convolutional Block Attention Module (CBAM), a self-designed light-weight target feature extraction network Ghost CBAM Network (GCNet) as the backbone of the whole model. Secondly, to improve the model multi-scale feature fusion, Path Aggregation Network (PAN) and Feature Pyramid Network (FPN) structures with the GhostConv module were used in neck net. Finally, for real-time semantic segmentation dairy farms multiple targets, a Segmentation Head (Seg Head), which is composed of the Receptive Field Block (RFB), Pyramid Pooling Module (PPM) and Feature Fusion Module (FFM), was introduced. Experimental results showed that the mAP@0.5 (mean average precision IoU = 0.5) of the dairy farm target reached 94.86%. Compared to the YOLOv5 model, the precision and recall was improved by 7.47% and 6.85%, respectively. In comparison to the YOLOv7 model, the precision was improved by 5.1%. Furthermore, when compared to the SSD model, the proposed model have reduced the number of model parameters by 92.43%, and its average detection time was reduced by 84.37 ms, which is ideal for meeting the real-time target recognition requirements. The average detection time of the model is 66.43 ms, making it more suitable for deployment in embedded devices. Compared with Ghost CBAM-Detection (GC-Detect) without the introduction of the Seg Head, the precision, recall and mAP@0.5 was improved by 4.49%, 4.92% and 6.58%, respectively. The research results can provide accurate algorithms for real-time and efficient identification of dairy farm targets for pushing robots, and provide more effective road and environmental scene segmentation methods for autonomous walking.

Learning Interpretable Microscopic Features of Tumor by Multi-task Adversarial CNNs To Improve Generalization

Adopting Convolutional Neural Networks (CNNs) in the daily routine of primary diagnosis requires not only near-perfect precision, but also a sufficient degree of generalization to data acquisition shifts and transparency. Existing CNN models act as black boxes, not ensuring to the physicians that important diagnostic features are used by the model. Building on top of successfully existing techniques such as multi-task learning, domain adversarial training and concept-based interpretability, this paper addresses the challenge of introducing diagnostic factors in the training objectives. Here we show that our architecture, by learning end-to-end an uncertainty-based weighting combination of multi-task and adversarial losses, is encouraged to focus on pathology features such as density and pleomorphism of nuclei, e.g. variations in size and appearance, while discarding misleading features such as staining differences. Our results on breast lymph node tissue show significantly improved generalization in the detection of tumorous tissue, with best average AUC 0.89 (0.01) against the baseline AUC 0.86 (0.005). By applying the interpretability technique of linearly probing intermediate representations, we also demonstrate that interpretable pathology features such as nuclei density are learned by the proposed CNN architecture, confirming the increased transparency of this model. This result is a starting point towards building interpretable multi-task architectures that are robust to data heterogeneity. Our code is available at &lt;a href='https://github.com/maragraziani/multitask_adversarial'&gt;https://github.com/maragraziani/multitask_adversarial&lt;/a&gt;