Multi-task Generative Adversarial Network Research Articles

STAGE: a spatiotemporal-knowledge enhanced multi-task generative adversarial network (GAN) for trajectory generation

Individual trajectory data play a pivotal role in various application fields, such as urban planning, traffic control, and epidemic simulation. Despite the diverse means for data collection in current times, the real-world trajectory data in practical application remains severely limited due to concerns over personal privacy. In this study, we designed a Spatiotemporal-knowledge enhanced multi-TAsk GEnerative adversarial network (GAN), named STAGE, to generate synthetic trajectories that statistically resemble the real data without recycling personal information. In STAGE, we designed a multi-task generator with three stages of spatio-temporal generation tasks, i.e. activity-sequence generation task, township-level trajectory generation task, and neighborhood-level trajectory generation task, with the last one as the main task while the other two as auxiliary tasks. Meanwhile, we designed a spatial consistency loss in the adversarial training process to assess the spatial consistency of generated trajectories at different spatial scales. Experiment results show that compared to the baselines, trajectories generated by our method have closer data distributions to the real ones. We argued that the designs of spatiotemporal-knowledge enhanced generation tasks and training loss benefit the spatiotemporal generation processes, which help reproduce the temporal patterns of human daily activities and spatial distribution of human movements.

Multi-task generative adversarial network for retinal optical coherence tomography image denoising

Objective. Optical coherence tomography (OCT) has become an essential imaging modality for the assessment of ophthalmic diseases. However, speckle noise in OCT images obscures subtle but important morphological details and hampers its clinical applications. In this work, a novel multi-task generative adversarial network (MGAN) is proposed for retinal OCT image denoising. Approach. To strengthen the preservation of retinal structural information in the OCT denoising procedure, the proposed MGAN integrates adversarial learning and multi-task learning. Specifically, the generator of MGAN simultaneously undertakes two tasks, including the denoising task and the segmentation task. The segmentation task aims at the generation of the retinal segmentation map, which can guide the denoising task to focus on the retina-related region based on the retina-attention module. In doing so, the denoising task can enhance the attention to the retinal region and subsequently protect the structural detail based on the supervision of the structural similarity index measure loss. Main results. The proposed MGAN was evaluated and analyzed on three public OCT datasets. The qualitative and quantitative comparisons show that the MGAN method can achieve higher image quality, and is more effective in both speckle noise reduction and structural information preservation than previous denoising methods. Significance. We have presented a MGAN for retinal OCT image denoising. The proposed method provides an effective way to strengthen the preservation of structural information while suppressing speckle noise, and can promote the OCT applications in the clinical observation and diagnosis of retinopathy.

Multitask GANs for Oil Spill Classification and Semantic Segmentation Based on SAR Images

The increasingly frequent marine oil spill disasters have great harm to the marine ecosystem. As an essential means of remote sensing monitoring, synthetic aperture radar (SAR) images can detect oil spills in time and reduce marine pollution. Many look-alike oil spill regions are difficult to distinguish in SAR images, and the scarcity of real oil spill data makes it difficult for deep learning networks to train effectively. In order to solve the above problems, this paper designs a multi-task generative adversarial networks (MTGANs) oil spill detection model to distinguish oil spills and look-alike oil spills and segment oil spill areas in one framework. The discriminator of the first GAN is transformed into a classifier, which can effectively distinguish between real and look-alike oil spills. The generator of the second GAN model integrates a fully convolutional symmetric structure and multiple convolution blocks. Multiple convolution blocks can extract the shallow oil spill information, and the fully convolutional symmetric structure can extract the deeper features of the oil spill information. The algorithm only needs to use a small number of oil spill images as the training set to train the network, and the limitation of the oil spill dataset can be solved. Validation evaluations are conducted on three datasets of Sentinel-1, ERS-1/2 and GF-3 satellites, and the experimental results demonstrate that the proposed MTGANs oil spill detection framework outperforms other models in oil spill classification and semantic segmentation. Among them, the classification accuracy of the oil spill and look-alikes can reach 97.22 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\%$</tex-math></inline-formula> . The average OA for semantic segmentation of the oil spill area can be 97.47 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\%$</tex-math></inline-formula> and the average precision can reach 86.69 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\%$</tex-math></inline-formula> . The code of this work will be available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/fjc1575/Marine-Oil-Spill/tree/main/MTGANs</uri> for the sake of reproducibility.

Surface Defect Detection Method Based on Improved Semisupervised Multitask Generative Adversarial Network

The detection methods based on deep learning networks have attracted widespread interest in industrial manufacture. However, the existing methods are mainly trapped by a large amount of training data with excellent labels and also show difficulty for the simultaneous detection of multiple defects in practical detection. Therefore, in this article, a defect detection method based on improved semisupervised multitask generative adversarial network (iSSMT-GAN) is proposed for generating better image features and improving classification accuracy. First, the training data are manually labeled according to the types of defects, and the generative adversarial network (GAN) is constructed according to the reliable annotations about defects. Thus, a classification decision surface for the detection of multitype defects is formed in the discriminative network of GAN in an integrated manner. Moreover, the semisupervised samples generated by the discriminative network give the generative network feedback for enhancing the image features and avoiding gradient disappearance or overfitting. Finally, the experimental results show that the proposed method can generate high-quality image features compared with the classic GAN. Furthermore, this increase in classification accuracy of RegNet model, MobileNet v3 model, VGG-19 model, and AlexNet-based transfer learning is 3.13%, 2.30%, 2.48%, and 3.12%, respectively.

Multitask GANs for Semantic Segmentation and Depth Completion With Cycle Consistency.

Semantic segmentation and depth completion are two challenging tasks in scene understanding, and they are widely used in robotics and autonomous driving. Although several studies have been proposed to jointly train these two tasks using some small modifications, such as changing the last layer, the result of one task is not utilized to improve the performance of the other one despite that there are some similarities between these two tasks. In this article, we propose multitask generative adversarial networks (Multitask GANs), which are not only competent in semantic segmentation and depth completion but also improve the accuracy of depth completion through generated semantic images. In addition, we improve the details of generated semantic images based on CycleGAN by introducing multiscale spatial pooling blocks and the structural similarity reconstruction loss. Furthermore, considering the inner consistency between semantic and geometric structures, we develop a semantic-guided smoothness loss to improve depth completion results. Extensive experiments on the Cityscapes data set and the KITTI depth completion benchmark show that the Multitask GANs are capable of achieving competitive performance for both semantic segmentation and depth completion tasks.

Joint segmentation and classification task via adversarial network: Application to HEp-2 cell images

The recognition of epithelial type 2 (HEp-2) cells has become an important tool for detecting autoimmune diseases diagnosis by using indirect immunofluorescence (IIF) images. For many medical image tasks, the accurate segmentation is considered as the primary step for the classification task, which inspires researchers to solve problems by jointly implementing multiple tasks. For this reason, we devise a hybrid network architecture, which combines the segmentation and classification networks for the final classification of HEp-2 cells, where a multi-task generative adversarial networks (GANs) is employed to produce accurate segmentation masks for improving the latter classification performance. Specifically, the devised generation and discrimination subnetworks establish the GANs architecture for the accurate segmentation masks of HEp-2 cells. Also, the original images are utilized as the conditional input, which are concatenated by the generated masks and the ground-truth to train the discriminator for two tasks: the first task determines whether the generated mask is a ground-truth or not and the other one distinguishes the category of the processed HEp-2 cell image. Furthermore, the ResNet-34 and MobileNetv3 are used as segmentation and classification base network, respectively. We modify the MobileNetv3 structure by adding the channel of the middle outputs, which is called augmented channel MobileNetv3 (ACM-Net). Both the discriminator and classifier share the weights of ACM-Net. The extensive experiments on the public ICPR 2016 task1 dataset show that the proposed hybrid-task based GANs structure can obtain promising segmentation and classification performance via jointly training mode, which achieves a Dice of 97.04% and for segmentation and a prediction accuracy of 98.82% for classification, respectively.

Collaborative image compression and classification with multi-task learning for visual Internet of Things

Widespread deployment of the Internet of Things (IoT) has changed the way that network services are developed, deployed, and operated. Most onboard advanced IoT devices are equipped with visual sensors that form the so-called visual IoT. Typically, the sender would compress images, and then through the communication network, the receiver would decode images, and then analyze the images for applications. However, image compression and semantic inference are generally conducted separately, and thus, current compression algorithms cannot be transplanted for the use of semantic inference directly. A collaborative image compression and classification framework for visual IoT applications is proposed, which combines image compression with semantic inference by using multi-task learning. In particular, the multi-task Generative Adversarial Networks (GANs) are described, which include encoder, quantizer, generator, discriminator, and classifier to conduct simultaneously image compression and classification. The key to the proposed framework is the quantized latent representation used for compression and classification. GANs with perceptual quality can achieve low bitrate compression and reduce the amount of data transmitted. In addition, the design in which two tasks share the same feature can greatly reduce computing resources, which is especially applicable for environments with extremely limited resources. Using extensive experiments, the collaborative compression and classification framework is effective and useful for visual IoT applications.

Centered Multi-Task Generative Adversarial Network for Small Object Detection.

Despite the breakthroughs in accuracy and efficiency of object detection using deep neural networks, the performance of small object detection is far from satisfactory. Gaze estimation has developed significantly due to the development of visual sensors. Combining object detection with gaze estimation can significantly improve the performance of small object detection. This paper presents a centered multi-task generative adversarial network (CMTGAN), which combines small object detection and gaze estimation. To achieve this, we propose a generative adversarial network (GAN) capable of image super-resolution and two-stage small object detection. We exploit a generator in CMTGAN for image super-resolution and a discriminator for object detection. We introduce an artificial texture loss into the generator to retain the original feature of small objects. We also use a centered mask in the generator to make the network focus on the central part of images where small objects are more likely to appear in our method. We propose a discriminator with detection loss for two-stage small object detection, which can be adapted to other GANs for object detection. Compared with existing interpolation methods, the super-resolution images generated by CMTGAN are more explicit and contain more information. Experiments show that our method exhibits a better detection performance than mainstream methods.

Single-image shadow removal using detail extraction and illumination estimation

Deep learning-based shadow removal methods are frequently hard to obtain a detail-rich and boundary-smoothing shadow removal result. In this work, we propose an illumination-sensitive filter and a multi-task generative adversarial networks architecture to tackle these problems. Firstly, we detect the shadow for the input shadow image and use the illumination-sensitive filter to extract the texture information for generating a coarse image with fewer texture details. Secondly, we conduct illumination estimation for this coarse shadow image to remove the shadow indirectly. Next, we restore the shadow boundary realistically inspired by the idea of image in painting. Finally, we recover the texture details for obtaining the final shadow removal result. Besides, we filter two large benchmark datasets, i.e., SRD and ISTD, to create a Low Error Synthesized Dataset (LESD). The extensive experiments demonstrate that our method can achieve superior performance to state of the arts.

GENERATING SYNTHETIC TRAINING DATA FOR OBJECT DETECTION USING MULTI-TASK GENERATIVE ADVERSARIAL NETWORKS

Abstract. Nowadays, digitizing roadside objects, for instance traffic signs, is a necessary step for generating High Definition Maps (HD Map) which remains as an open challenge. Rapid development of deep learning technology using Convolutional Neural Networks (CNN) has achieved great success in computer vision field in recent years. However, performance of most deep learning algorithms highly depends on the quality of training data. Collecting the desired training dataset is a difficult task, especially for roadside objects due to their imbalanced numbers along roadside. Although, training the neural network using synthetic data have been proposed. The distribution gap between synthetic and real data still exists and could aggravate the performance. We propose to transfer the style between synthetic and real data using Multi-Task Generative Adversarial Networks (SYN-MTGAN) before training the neural network which conducts the detection of roadside objects. Experiments focusing on traffic signs show that our proposed method can reach mAP of 0.77 and is able to improve detection performance for objects whose training samples are difficult to collect.

Classification of Hyperspectral Images via Multitask Generative Adversarial Networks

Deep learning has shown its huge potential in the field of hyperspectral image (HSI) classification. However, most of the deep learning models heavily depend on the quantity of available training samples. In this article, we propose a multitask generative adversarial network (MTGAN) to alleviate this issue by taking advantage of the rich information from unlabeled samples. Specifically, we design a generator network to simultaneously undertake two tasks: the reconstruction task and the classification task. The former task aims at reconstructing an input hyperspectral cube, including the labeled and unlabeled ones, whereas the latter task attempts to recognize the category of the cube. Meanwhile, we construct a discriminator network to discriminate the input sample coming from the real distribution or the reconstructed one. Through an adversarial learning method, the generator network will produce real-like cubes, thus indirectly improving the discrimination and generalization ability of the classification task. More importantly, in order to fully explore the useful information from shallow layers, we adopt skip-layer connections in both reconstruction and classification tasks. The proposed MTGAN model is implemented on three standard HSIs, and the experimental results show that it is able to achieve higher performance than other state-of-the-art deep learning models.

Multi-task Generative Adversarial Network for Detecting Small Objects in the Wild

Object detection results have been rapidly improved over a short period of time with the development of deep convolutional neural networks. Although impressive results have been achieved on large/medium sized objects, the performance on small objects is far from satisfactory and one of remaining open challenges is detecting small object in unconstrained conditions (e.g. COCO and WIDER FACE benchmarks). The reason is that small objects usually lack sufficient detailed appearance information, which can distinguish them from the backgrounds or similar objects. To deal with the small object detection problem, in this paper, we propose an end-to-end multi-task generative adversarial network (MTGAN), which is a general framework. In the MTGAN, the generator is a super-resolution network, which can up-sample small blurred images into fine-scale ones and recover detailed information for more accurate detection. The discriminator is a multi-task network, which describes each inputted image patch with a real/fake score, object category scores, and bounding box regression offsets. Furthermore, to make the generator recover more details for easier detection, the classification and regression losses in the discriminator are back-propagated into the generator during training process. Extensive experiments on the challenging COCO and WIDER FACE datasets demonstrate the effectiveness of the proposed method in restoring a clear super-resolved image from a blurred small one, and show that the detection performance, especially for small sized objects, improves over state-of-the-art methods by a large margin.

No-Reference Image Quality Assessment Based on Multi-Task Generative Adversarial Network

Since human observers are the ultimate receivers of an image, most of the image quality assessment (IQA) methods are based on analysis of the properties and mechanism of the human visual system. However, due to the lack of undistorted images for a reference, the accuracy of the no-reference IQA (NR-IQA) cannot compete with that of the full-reference IQA (FR-IQA). To bridge the performance gap between the FR-IQA and NR-IQA methods, we propose a NR-IQA method based on multi-task generative adversarial network, which attempts to restore dependable hallucinated images to compensate for the missing corresponding reference images. Two tasks, hallucination images and the quality maps are outputted by the generator and are combined with the specific loss to improve the reliability of hallucination images. Besides, two discriminator networks are used to respectively distinguish the undistorted images and hallucination images pairs, quality maps and structural similarity index measurement maps pairs. Finally, the hallucination images and distorted images are input into the IQA network, and quality scores are evaluated based on the differences between them. The superiority of our proposed method is verified by several different experiments on the LIVE datasets, TID2008 datasets, and TID2013 datasets.

Multi-Task GANs for View-Specific Feature Learning in Gait Recognition

Gait recognition is of great importance in the fields of surveillance and forensics to identify human beings since gait is the unique biometric feature that can be perceived efficiently at a distance. However, the accuracy of gait recognition to some extent suffers from both the variation of view angles and the deficient gait templates. On one hand, the existing cross-view methods focus on transforming gait templates among different views, which may accumulate the transformation error in a large variation of view angles. On the other hand, a commonly used gait energy image template loses temporal information of a gait sequence. To address these problems, this paper proposes multi-task generative adversarial networks (MGANs) for learning view-specific feature representations. In order to preserve more temporal information, we also propose a new multi-channel gait template, called period energy image (PEI). Based on the assumption of view angle manifold, the MGANs can leverage adversarial training to extract more discriminative features from gait sequences. Experiments on OU-ISIR, CASIA-B, and USF benchmark data sets indicate that compared with several recently published approaches, PEI + MGANs achieves competitive performance and is more interpretable to cross-view gait recognition.

Illumination-Aware Multi-Task GANs for Foreground Segmentation

Foreground-background segmentation has been an active research area over the years. However, conventional models fail to produce accurate results when challenged with the videos of challenging illumination conditions. In this paper, we present a robust model that allows accurately extracting the foreground even in exceptionally dark or bright scenes and in continuously varying illumination in a video sequence. This is accomplished by a triple multi-task generative adversarial network (TMT-GAN) that effectively models the semantic relationship between the dark and bright images and performs binary segmentation end-to-end. Our contribution is twofold: first, we show that by jointly optimizing the GAN loss and the segmentation loss, our network simultaneously learns both tasks that mutually benefit each other. Second, fusing features of images with varying illumination into the segmentation branch vastly improve the performance of the network. Comparative evaluations on highly challenging real and synthetic benchmark datasets (ESI and SABS) demonstrate the robustness of TMT-GAN and its superiority over state-of-the-art approaches.