Adversarial Learning Techniques Research Articles

Multiscale and Adversarial Learning-Based Semi-Supervised Semantic Segmentation Approach for Crack Detection in Concrete Structures

Typically, the operational lifetime of underground concrete structures is several decades. At present, many such structures are approaching their original life expectancy. In this stage, the essential functionality of the structures may be considerably degraded, leading to various safety hazards such as collapse roof and tunnel flooding. In general, to overcome such problems, the maintenance of underground structures has been conducted through manual subjective inspections so far. However, recently, several objective inspection technologies have been actively developed by fusing artificial intelligence and imaging techniques recently. In particular, deep learning algorithms have been developed to detect concrete cracks, based on a large amount of data for supervised learning, including numerous labeled images. Such data acquisition requires considerable time and effort. To reduce these costs, in this study, multiscale and adversarial learning techniques were applied to realize crack detection. A total of 1,200 labeled data and 3,000 unlabeled data were used to implement and verify the proposed method. The multiscale segmentation neural network, discriminator neural network, and adversarial learning technique were used to realize accurate crack detection, enhance the learning performance, and ensure the efficiency of training data, respectively. The resulting algorithm had a pixel accuracy, mean intersection over union, frequency weighted intersection over union, and F1 score of 98.176%, 88.936%, 96.525%, and 88.789%, respectively. The proposed technique can be used to examine the conditions to ensure the safe maintenance of aging structures.

HMTL: Heterogeneous Modality Transfer Learning for Audio-Visual Sentiment Analysis

Multimodal sentiment analysis is an extended approach to traditional language-based sentiment analysis, which uses other relevant modality data. Multimodal sentiment analysis usually applies visual, textual, and acoustic representations for sentiment prediction. Recently, various data fusion methodologies have been proposed for multimodal sentiment analysis. In most cases, textual modality plays a major role, and visual and acoustic modalities are used as auxiliary sources for multimodal sentiment analysis. However, in general multimedia such as video, text transcripts of an individual's speech are not provided. Research on an audio-visual sentiment analysis methodology that does not depend on text modality is essential for multimodal sentiment analysis in real-world industrial applications. Therefore, it is important to improve audio-visual sentiment analysis because it currently exhibits lower performance than multimodal sentiment analysis, including text modality. In this paper, we propose heterogeneous modality transfer learning (HMTL) to utilize the knowledge of aligned text data as a source modality in transfer learning to improve audio-visual sentiment analysis performance. Our approach uses a decoder and adversarial learning techniques to reduce the gap between the source and target modalities in the embedded space for multimodal representation. Our proposed methodology experimentally outperformed recent unimodal and bimodal audio-visual sentiment analysis achievements.

Image generation from bounding box-represented semantic labels

Image generation with pixel-wise semantic information is suitable for the development of adversarial learning techniques. In this study, we propose a method for synthesizing objects with class-specific textures and fine-scale details based on bounding box-represented semantic labels. To achieve this goal, we note that the traditional generative adversarial network (GAN) uses noise as an input to generate realistic images with sufficient textures and details, but it cannot be guided by specific targets and requirements. By contrast, conditional GAN (cGAN) can involve various types of guiding information but it often ignores specific textures and details, thereby leading to less realistic results and low resolution. Thus, we propose a new translator-enhancer framework by combining cGAN and GAN to achieve high quality image generation. cGAN is used as a translator to match the semantic constraints whereas GAN is employed as an enhancer to provide details and textures. We also propose a new form of semantic label map as an input, which is represented by instance-level bounding boxes rather than segmentation masks. The semantic label map represented by bounding boxes makes it easier for users to provide the inputs and it also gives greater flexibility when generating object boundaries. The results obtained from qualitative and quantitative experiments showed that our method can generate realistic images of objects with semantic labels represented by bounding boxes. Our method can be used to generate images of novel scenes to support learning tasks during training with various scenes, which are difficult to capture in the real world.