Multi-task Convolutional Neural Network Research Articles

Robust face recognition based on multi-task convolutional neural network.

Due to the lack of prior knowledge of face images, large illumination changes, and complex backgrounds, the accuracy of face recognition is low. To address this issue, we propose a face detection and recognition algorithm based on multi-task convolutional neural network (MTCNN). In our paper, MTCNN mainly uses three cascaded networks, and adopts the idea of candidate box plus classifier to perform fast and efficient face recognition. The model is trained on a database of 50 faces we have collected, and Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measurement (SSIM), and receiver operating characteristic (ROC) curve are used to analyse MTCNN, Region-CNN (R-CNN) and Faster R-CNN. The average PSNR of this technique is 1.24 dB higher than that of R-CNN and 0.94 dB higher than that of Faster R-CNN. The average SSIM value of MTCNN is 10.3% higher than R-CNN and 8.7% higher than Faster R-CNN. The Area Under Curve (AUC) of MTCNN is 97.56%, the AUC of R-CNN is 91.24%, and the AUC of Faster R-CNN is 92.01%. MTCNN has the best comprehensive performance in face recognition. For the face images with defective features, MTCNN still has the best effect. This algorithm can effectively improve face recognition to a certain extent. The accuracy rate and the reduction of the false detection rate of face detection can not only be better used in key places, ensure the safety of property and security of the people, improve safety, but also better reduce the waste of human resources and improve efficiency.

Social Group Behavior Analysis Model Integrating Multitask Learning and Convolutional Neural Network

Social group behavior analysis has always been the key research direction of sociologists and psychologists. With the rapid development of the Internet of Things and the proposal of deep learning theory, convolutional neural networks are also used in social group research. In the process of social development, group incidents continue to increase, and there are more and more studies on social group behavior analysis. Although the research content and research methods are also richer, the research that combines the Internet of Things, convolutional neural network, and group behavior is more. This article will specifically propose a social group behavior analysis model that combines multitask learning and convolutional neural networks. This paper deeply learns the research of convolutional neural network and group behavior‐related theories, makes full use of the advantages of convolutional neural network algorithm and multitask learning mode, and builds a social group behavior analysis model based on multitask learning and convolutional neural network. The experimental results on different data sets are analyzed. The results show that the accuracy rate of the experimental algorithm of convolutional neural network is as high as 95.10%, and it is better than other algorithms in time complexity, which is very suitable for social group behavior analysis.

Multi-task Convolution Neural Network for Season-Insensitive Chlorophyll-A Estimation in Inland Water

Chlorophyll-a (Chl-a) concentration, a crucial indicator of phytoplankton biomass, is sensitive to seasonality. The variations in trophic states regarding seasonality and the changes of spectral properties of water bodies pose uncertainties to the accuracy of remote sensing semiempirical models. In particular, lakes in subtropical regions generally experience different trophic states in dry and wet seasons. In this study, a season-insensitive Chl-a retrieval model using multitask convolution neural network with multiple output layers (MCNN) is proposed. A layer-sharing network combined with data augmentation is adopted to alleviate the issue of insufficient quantity of in situ samples. In addition, a hyperparameter optimization is performed to automatically refine the MCNN architecture. To evaluate the accuracy of proposed method, Laguna Lake, one of the largest lakes in Southeast Asia, is selected as the validation target. The lake is characterized by oligotrophic and mesotrophic states in wet season, whereas the states change to mesotrophic and low-level eutrophic states in dry season. A collection of Sentinel-3 Ocean and Land Colour Instrument Level-2 images and 409 in situ samples with the Chl-a concentration range 1.24–22.30 mg $\cdot$ m $^{-3}$ were used for model calibration and evaluation. Experimental results showed that MCNN with the performance of average $\boldsymbol{R^{2}}$ = 0.74, RMSE = 2.06 mg $\cdot$ m $^{-3}$ , Pearson's $\boldsymbol {r}$ = 0.86 outperforms related semiempirical models, including normalized difference chlorophyll index, two-band and three-band models, and WaterNet. The Chl-a prediction accuracy was improved by 7.19–14.6%, in terms of RMSE, compared with WaterNet.

Face anti-spoofing algorithm combined with CNN and brightness equalization

Face anti-spoofing is a relatively important part of the face recognition system, which has great significance for financial payment and access control systems. Aiming at the problems of unstable face alignment, complex lighting, and complex structure of face anti-spoofing detection network, a novel method is presented using a combination of convolutional neural network and brightness equalization. Firstly, multi-task convolutional neural network (MTCNN) based on the cascade of three convolutional neural networks (CNNs), P-net, R-net, and O-net are used to achieve accurate positioning of the face, and the detected face bounding box is cropped by a specified multiple, then brightness equalization is adopted to perform brightness compensation on different brightness areas of the face image. Finally, data features are extracted and classification is given by utilizing a 12-layer convolution neural network. Experiments of the proposed algorithm were carried out on CASIA-FASD. The results show that the classification accuracy is relatively high, and the half total error rate (HTER) reaches 1.02%.

Multi-face recognition and dynamic tracking based on reinforcement learning algorithm

Aiming at the problem that the current low accuracy rate of face detection and target tracking, a reinforcement learning algorithm is proposed, which integrates face detection technology and target tracking technology organically, adopts the face detection algorithm based on Multi-Task Convolutional Neural Network (MTCNN) and target tracking algorithm based on Kalman filtering, so as to realize face detection, multiplayer face recognition and dynamic tracking of personnel movement. In this paper, the configuration environment is Anaconda, the operating platform is PyCharm, the video-based face detection and dynamic capture and rapid identification system has been designed and developed. The system consists of two modules: face detection module and target tracking module. The optimized face detection and dynamic capture algorithm improved the detection success rate by about 11.5%, the face detection success rate by about 15.2%, the dynamic capture success rate increased by about 12.0%, and the optimized system has a wider practicality.

A Multi-Task CNN for Maritime Target Detection

In this letter, we construct MaRine ShiP (MRSP-13), a novel dataset containing 37,161 ship target images belonging to 13 classes with bounding box annotation, and among them there are 3051 images labeled with pixel-level annotation. This dataset equips us with the capability to conduct baseline experiments on maritime target classification, detection and segmentation. We propose a cross-layer multi-task CNN model for maritime target detection, which can simultaneously solve ship target detection, classification, and segmentation. Experimental results have demonstrated the efficiency of the MRSP-13 dataset to be used for maritime target analysis. In addition, the results validate the fact that by adopting the strategies of feature sharing, joint learning, and cross-layer connections, the proposed model achieves superior performance with less annotations. We believe that our MRSP-13 dataset and corresponding baseline experiments will lay down the foundation for further research in maritime target processing.

DeLFIQE—A Low-Complexity Deep Learning-Based Light Field Image Quality Evaluator

Light field image quality assessment (LF-IQA) has attracted increasing research interests due to the fast-growing demands for immersive media experience. The existing LF-IQA metrics, however, heavily rely on high-complexity statistics-based feature extraction, which is computationally intensive and hardly sustainable in real-time systems. In this article, we propose a low-complexity deep learning-based light field image quality evaluator (DeLFIQE) that achieves both improved accuracy and significantly reduced complexity in measuring LF image quality. On one hand, we propose the novel discriminative epipolar plane image (D-EPI) patches, which reduces the high-complexity 4-D light field IQA processing to lower dimension and lower complexity 2-D IQA processing. On the other hand, we design a distortion-aware multitask learning scheme in DeLFIQE to improve the quality measurement accuracy and generalization. Specifically, alongside the main task that predicts the light field image (LFI) quality, we design an auxiliary classification task that classifies the D-EPI patches based on their distortion types and severity levels, which we believe both are key factors affecting the LFI quality. To the best of our knowledge, this is the first work in LF-IQA with a dedicatedly designed deep multitask convolutional neural network (CNN) model. Our extensive experimental results show that DeLFIQE outperforms the state-of-the-art metrics on representative benchmark datasets with significantly reduced complexity.

Learning location and seabed type from a moving mid-frequency source.

While source localization and seabed classification are often approached separately, the convolutional neural networks (CNNs) in this paper simultaneously predict seabed type, source depth and speed, and the closest point of approach. Different CNN architectures are applied to mid-frequency tonal levels from a moving source recorded on a 16-channel vertical line array (VLA). After training each CNN on synthetic data, a statistical representation of predictions on test cases is presented. The performance of a single regression-based CNN is compared to a multitask CNN in which regression is used for the source parameters and classification for the seabed type. The impact of water sound speed profile and seabed variations on the predictions is evaluated using simulated test cases. Environmental mismatch between the training and testing data has a negative impact on source depth estimates, while the remaining labels are estimated tolerably well but with a bias towards shorter ranges. Similar results are found for data measured on two VLAs during Seabed Characterization Experiment 2017. This work shows the superiority of multitask learning and the potential for using a CNN to localize an acoustic source and detect the surficial seabed properties from mid-frequency sounds.

Multi-Scale and Multi-Channel Networks for CSI Feedback in Massive MIMO System

In the frequency division duplex (FDD) mode of the massive MIMO system, the system needs to perform coding through channel state information (CSI) to obtain performance gains. However, the number of antennas of the base station has been greatly increased, resulting in a rapid increase in the overhead for the user terminal to feedback CSI to the base station. In this article, we propose a method based on multi-task CNN to achieve compression and reconstruction of channel state information through a multi-scale and multi-channel convolutional neural network. We also introduce a dynamic learning rate model to improve the accuracy of channel state information reconstruction. The simulation results show that compared with the original CsiNet and other work, the proposed CSI feedback network has better reconstruction performance.

Detecting High-Engaging Breaking News Rumors in Social Media

Users from all over the world increasingly adopt social media for newsgathering, especially during breaking news. Breaking news is an unexpected event that is currently developing. Early stages of breaking news are usually associated with lots of unverified information, i.e., rumors. Efficiently detecting and acting upon rumors in a timely fashion is of high importance to minimize their harmful effects. Yet, not all rumors have the potential to spread in social media. High-engaging rumors are those written in a manner that ensures achievement of the highest prevalence among the recipients. They are difficult to detect, spread very fast, and can cause serious damage to society. In this article, we propose a new multi-task Convolutional Neural Network (CNN) attention-based neural network architecture to jointly learn the two tasks of breaking news rumors detection and breaking news rumors popularity prediction in social media. The proposed model learns the salient semantic similarities among important features for detecting high-engaging breaking news rumors and separates them from the rest of the input text. Extensive experiments on five real-life datasets of breaking news suggest that our proposed model outperforms all baselines and is capable of detecting breaking news rumors and predicting their future popularity with high accuracy.

LaeNet: A Novel Lightweight Multitask CNN for Automatically Extracting Lake Area and Shoreline from Remote Sensing Images

Variations of lake area and shoreline can indicate hydrological and climatic changes effectively. Accordingly, how to automatically and simultaneously extract lake area and shoreline from remote sensing images attracts our attention. In this paper, we formulate lake area and shoreline extraction as a multitask learning problem. Different from existing models that take the deep and complex network architecture as the backbone to extract feature maps, we present LaeNet—a novel end-to-end lightweight multitask fully CNN with no-downsampling to automatically extract lake area and shoreline from remote sensing images. Landsat-8 images over Selenco and the vicinity in the Tibetan Plateau are utilized to train and evaluate our model. Experimental results over the testing image patches achieve an Accuracy of 0.9962, Precision of 0.9912, Recall of 0.9982, F1-score of 0.9941, and mIoU of 0.9879, which align with the mainstream semantic segmentation models (UNet, DeepLabV3+, etc.) or even better. Especially, the running time of each epoch and the size of our model are only 6 s and 0.047 megabytes, which achieve a significant reduction compared to the other models. Finally, we conducted fieldwork to collect the in-situ shoreline position for one typical part of lake Selenco, in order to further evaluate the performance of our model. The validation indicates high accuracy in our results (DRMSE: 30.84 m, DMAE: 22.49 m, DSTD: 21.11 m), only about one pixel deviation for Landsat-8 images. LaeNet can be expanded potentially to the tasks of area segmentation and edge extraction in other application fields.

Visual Robot Relocalization Based on Multi-Task CNN and Image-Similarity Strategy.

The traditional CNN for 6D robot relocalization which outputs pose estimations does not interpret whether the model is making sensible predictions or just guessing at random. We found that convnet representations trained on classification problems generalize well to other tasks. Thus, we propose a multi-task CNN for robot relocalization, which can simultaneously perform pose regression and scene recognition. Scene recognition determines whether the input image belongs to the current scene in which the robot is located, not only reducing the error of relocalization but also making us understand with what confidence we can trust the prediction. Meanwhile, we found that when there is a large visual difference between testing images and training images, the pose precision becomes low. Based on this, we present the dual-level image-similarity strategy (DLISS), which consists of two levels: initial level and iteration-level. The initial level performs feature vector clustering in the training set and feature vector acquisition in testing images. The iteration level, namely, the PSO-based image-block selection algorithm, can select the testing images which are the most similar to training images based on the initial level, enabling us to gain higher pose accuracy in testing set. Our method considers both the accuracy and the robustness of relocalization, and it can operate indoors and outdoors in real time, taking at most 27 ms per frame to compute. Finally, we used the Microsoft 7Scenes dataset and the Cambridge Landmarks dataset to evaluate our method. It can obtain approximately 0.33 m and 7.51 accuracy on 7Scenes dataset, and get approximately 1.44 m and 4.83 accuracy on the Cambridge Landmarks dataset. Compared with PoseNet, our CNN reduced the average positional error by 25% and the average angular error by 27.79% on 7Scenes dataset, and reduced the average positional error by 40% and the average angular error by 28.55% on the Cambridge Landmarks dataset. We show that our multi-task CNN can localize from high-level features and is robust to images which are not in the current scene. Furthermore, we show that our multi-task CNN gets higher accuracy of relocalization by using testing images obtained by DLISS.

Multi-Task Convolutional Neural Network Leading to High Performance and Interpretability via Attribute Estimation

Multi-Task Convolutional Neural Network Leading to High Performance and Interpretability via Attribute Estimation

RGB2Hands

Tracking and reconstructing the 3D pose and geometry of two hands in interaction is a challenging problem that has a high relevance for several human-computer interaction applications, including AR/VR, robotics, or sign language recognition. Existing works are either limited to simpler tracking settings ( e.g. , considering only a single hand or two spatially separated hands), or rely on less ubiquitous sensors, such as depth cameras. In contrast, in this work we present the first real-time method for motion capture of skeletal pose and 3D surface geometry of hands from a single RGB camera that explicitly considers close interactions. In order to address the inherent depth ambiguities in RGB data, we propose a novel multi-task CNN that regresses multiple complementary pieces of information, including segmentation, dense matchings to a 3D hand model, and 2D keypoint positions, together with newly proposed intra-hand relative depth and inter-hand distance maps. These predictions are subsequently used in a generative model fitting framework in order to estimate pose and shape parameters of a 3D hand model for both hands. We experimentally verify the individual components of our RGB two-hand tracking and 3D reconstruction pipeline through an extensive ablation study. Moreover, we demonstrate that our approach offers previously unseen two-hand tracking performance from RGB, and quantitatively and qualitatively outperforms existing RGB-based methods that were not explicitly designed for two-hand interactions. Moreover, our method even performs on-par with depth-based real-time methods.

Detection of Head Raising Rate of Students in Classroom Based on Head Posture Recognition

The proliferation of smart mobile terminals has weakened the attention and reduced the learning efficiency of students, making them more likely to lower their heads. To quantify the classroom participation, it is helpful to detect the head raising rate (HRR) of students in classroom. To this end, this paper puts forward a novel method to recognize the HRR of students in classroom. Based on the map of predicted facial features, an extraction method was developed for the salient facial features of students, and used to realize model matching between facial contour and facial organ. Next, the face orientation of each student was determined by soft label coding. After that, a multi-task convolutional neural network (CNN) was constructed to detect the HRR of students. The authors also explained the regularization of the loss function, and the steps of target detection. The proposed method was proved effective through experiments. The research results provide a reference for the application of head posture recognition in other fields.

Abstract 14963: Accelerometer-measured Activity in Non-obstructive Hypertrophic Cardiomyopathy: Patient-generated Activity Measures Correlate With, and Are Convolutional Neural Network Predictors of, Clinical Parameters in the MAVERICK-HCM Study

Introduction: Overall activity characteristics for patients with hypertrophic cardiomyopathy (HCM) have not been quantified previously. The relationship between physical activity quantified by accelerometry and biomarkers, exercise capacity, and quality of life in patients with HCM is also unknown. Methods: MAVERICK-HCM was a double-blind, placebo-controlled, 16-week study of mavacamten in 59 patients with symptomatic non-obstructive HCM. Patients were asked to wear ActiGraph GT9X Link wrist-worn monitors for ≥11 days between screening and day 1, and between weeks 12 and 16. Features derived from raw accelerometry data included average daily accelerometer units (ADAU) and step count. Univariate Pearson correlation coefficients were calculated between accelerometry data and clinical parameters among all patients. A multi-task convolutional neural network (CNN) was trained on raw accelerometry datapoints to jointly predict clinical markers of HCM severity. Test and training sets were derived by randomly segmenting each patient’s triaxial accelerometry data into non-overlapping minute intervals. Results: Fifty patients wore the accelerometer for ≥1 compliant day. Mean wear time was 12 days during screening and 10 days during treatment. Activity measures are summarized and average step count was 3,076 steps at baseline ( Table ). Activity features correlated with peak oxygen uptake (pVO 2 ), log NT-proBNP, and KCCQ score ( Table ). CNN predictions of clinical measures from activity data found Spearman R correlations of 0.82 for pVO 2 , 0.92 for log NT-proBNP, 0.82 for KCCQ, and 0.79 for E/e’. Conclusions: HCM patients in the MAVERICK study averaged only 3,000 steps/day. Markers of physical activity drawn from accelerometry are associated with standard clinical markers of HCM severity. Deep learning models can be constructed to predict markers of HCM severity from patients’ raw accelerometry data.

Image Aesthetics Assessment Based on Multi-stream CNN Architecture and Saliency Features

ABSTRACT In this paper, we explore how higher-level perceptual information based on visual attention can be used for aesthetic assessment of images. We assume that visually dominant subjects in a photograph influence stronger aesthetic interest. In other words, visual attention may be a key to predicting photographic aesthetics. Our proposed aesthetic assessment method, which is based on multi-stream and multi-task convolutional neural networks (CNNs), extracts global features and saliency features from an input image. These provide higher-level visual information such as the quality of the photo subject and the subject–background relationship. Results from our experiments support the effectiveness of our approach.

Parallel Multi-task Cascade Convolution Neural Network Optimization Algorithm for Real-time Dynamic Face Recognition

Parallel Multi-task Cascade Convolution Neural Network Optimization Algorithm for Real-time Dynamic Face Recognition

A Multitask Cascading CNN with MultiScale Infrared Optical Flow Feature Fusion-Based Abnormal Crowd Behavior Monitoring UAV.

Visual-based object detection and understanding is an important problem in computer vision and signal processing. Due to their advantages of high mobility and easy deployment, unmanned aerial vehicles (UAV) have become a flexible monitoring platform in recent years. However, visible-light-based methods are often greatly influenced by the environment. As a result, a single type of feature derived from aerial monitoring videos is often insufficient to characterize variations among different abnormal crowd behaviors. To address this, we propose combining two types of features to better represent behavior, namely, multitask cascading CNN (MC-CNN) and multiscale infrared optical flow (MIR-OF), capturing both crowd density and average speed and the appearances of the crowd behaviors, respectively. First, an infrared (IR) camera and Nvidia Jetson TX1 were chosen as an infrared vision system. Since there are no published infrared-based aerial abnormal-behavior datasets, we provide a new infrared aerial dataset named the IR-flying dataset, which includes sample pictures and videos in different scenes of public areas. Second, MC-CNN was used to estimate the crowd density. Third, MIR-OF was designed to characterize the average speed of crowd. Finally, considering two typical abnormal crowd behaviors of crowd aggregating and crowd escaping, the experimental results show that the monitoring UAV system can detect abnormal crowd behaviors in public areas effectively.

Accelerated training of bootstrap aggregation-based deep information extraction systems from cancer pathology reports.

In machine learning, it is evident that the classification of the task performance increases if bootstrap aggregation (bagging) is applied. However, the bagging of deep neural networks takes tremendous amounts of computational resources and training time. The research question that we aimed to answer in this research is whether we could achieve higher task performance scores and accelerate the training by dividing a problem into sub-problems. The data used in this study consist of free text from electronic cancer pathology reports. We applied bagging and partitioned data training using Multi-Task Convolutional Neural Network (MT-CNN) and Multi-Task Hierarchical Convolutional Attention Network (MT-HCAN) classifiers. We split a big problem into 20 sub-problems, resampled the training cases 2,000 times, and trained the deep learning model for each bootstrap sample and each sub-problem-thus, generating up to 40,000 models. We performed the training of many models concurrently in a high-performance computing environment at Oak Ridge National Laboratory (ORNL). We demonstrated that aggregation of the models improves task performance compared with the single-model approach, which is consistent with other research studies; and we demonstrated that the two proposed partitioned bagging methods achieved higher classification accuracy scores on four tasks. Notably, the improvements were significant for the extraction of cancer histology data, which had more than 500 class labels in the task; these results show that data partition may alleviate the complexity of the task. On the contrary, the methods did not achieve superior scores for the tasks of site and subsite classification. Intrinsically, since data partitioning was based on the primary cancer site, the accuracy depended on the determination of the partitions, which needs further investigation and improvement. Results in this research demonstrate that 1. The data partitioning and bagging strategy achieved higher performance scores. 2. We achieved faster training leveraged by the high-performance Summit supercomputer at ORNL.