Style-transferred Images Research Articles

Non-parametric style transfer: Correlation-aware exact distribution matching

Early feed-forward neural methods of arbitrary image style transfer utilized the encoded feature map up to its second-order statistics, i.e., mean and variance (or covariance) of the encoded feature map. Recent methods have begun to utilize feature statistics beyond the 2nd-order statistics, e.g., a higher-order moment, slicing-based method, and histogram matching for better style representation. However, even in the distribution matching of the latest method, considering the channel-correlation and the order of duplicated values in the feature map is missing. This resulted in insufficient style transfer or artifacts on style-transferred images e.g., unexpected color or pattern co-occurrence, skewed color or style pattern into one side, noise between background and foreground, etc. In this work, we show how to make a feature transform layer to consider both channel-correlation and duplicated values in the feature map for exact feature distribution matching in style transfer. Our experimental results prove that the stylized images obtained with our method are more similar to the target style images qualitatively without losing content clearness and superior to the existing methods quantitatively in all possible style measures, e.g., 0.0311 lower sliced Wasserstein distance than the previous method and the most favored choice with 25.54% preference.

Street Object Detection from Synthesized and Processed Semantic Image: A Deep Learning Based Study

Semantic image synthesis plays an important role in the development of Advanced Driver Assistance System (ADAS). Street objects detection might be erroneous during raining or when images from vehicle’s camera are blurred, which can cause serious accidents. Therefore, automatic and accurate street object detection is a demanding research scope. In this paper, a deep learning based framework is proposed and investigated for street object detection from synthesized and processed semantic image. Firstly, a Conditional Generative Adversarial Network (CGAN) has been used to create the realistic image. The brightness of the CGAN generated image has been increased using neural style transfer method. Furthermore, Enhanced Super-Resolution Generative Adversarial Networks (ESRGAN) based image enhancement concept has been used to improve the resolution of style-transferred images. These processed images exhibit better clarity and high fidelity which is impactful in the performance improvement of object detector. Finally, the synthesized and processed images were used as input in a Region-based Convolutional Neural Network (Faster R-CNN) and a MobileNet Single Shot Detector (MobileNetSSDv2) model separately for object detection. The widely used Cityscape dataset is used to investigate the performance of the proposed framework. The results analysis shows that the used synthesized and processed input improves the performance of the detectors than the unprocessed counterpart. A comparison of the proposed detection framework with related state of the art techniques is also found satisfactory with a mean average precision (mAP) around 32.6%, whereas most of the cases, mAPs are reported in the range of 20–28% for this particular dataset.

Statistical image properties predict aesthetic ratings in abstract paintings created by neural style transfer.

Artificial intelligence has emerged as a powerful computational tool to create artworks. One application is Neural Style Transfer, which allows to transfer the style of one image, such as a painting, onto the content of another image, such as a photograph. In the present study, we ask how Neural Style Transfer affects objective image properties and how beholders perceive the novel (style-transferred) stimuli. In order to focus on the subjective perception of artistic style, we minimized the confounding effect of cognitive processing by eliminating all representational content from the input images. To this aim, we transferred the styles of 25 diverse abstract paintings onto 150 colored random-phase patterns with six different Fourier spectral slopes. This procedure resulted in 150 style-transferred stimuli. We then computed eight statistical image properties (complexity, self-similarity, edge-orientation entropy, variances of neural network features, and color statistics) for each image. In a rating study, we asked participants to evaluate the images along three aesthetic dimensions (Pleasing, Harmonious, and Interesting). Results demonstrate that not only objective image properties, but also subjective aesthetic preferences transferred from the original artworks onto the style-transferred images. The image properties of the style-transferred images explain 50 – 69% of the variance in the ratings. In the multidimensional space of statistical image properties, participants considered style-transferred images to be more Pleasing and Interesting if they were closer to a “sweet spot” where traditional Western paintings (JenAesthetics dataset) are represented. We conclude that NST is a useful tool to create novel artistic stimuli that preserve the image properties of the input style images. In the novel stimuli, we found a strong relationship between statistical image properties and subjective ratings, suggesting a prominent role of perceptual processing in the aesthetic evaluation of abstract images.

Joint generative and camera-aware clustering for unsupervised domain adaptation on person re-identification

Existing supervised person re-identification (Re-ID) methods demonstrate excellent performance. However, their performances suffer degradation when tested on an unseen different distributed domain. Generative adversarial network-based (GAN-based) and clustering- or pseudolabel-based methods are proposed to alleviate this problem. Due to the transfer scheme and the ignorance of correlations between the style-transferred and the original target images, the performance of GAN-based methods is unsatisfactory. We resolve these problems by jointly employing a generative strategy and performing camera-aware clustering for the target domain. Style-transferred images are generated from source cameras to target cameras, and then they are merged into the target domain selectively after exploiting their domain-specific discriminative information. To reduce the noise in generated images, we propose a domain-level boundary separation loss to group the transferred images and push them away from the original target images. The camera-level neighborhood-based clustering is proposed to learn well-clustered features in a camera-aware manner. Extensive experiments on two commonly used person Re-ID datasets demonstrate that our proposed method can achieve state-of-the-art performance.

Dermoscopic Image Classification with Neural Style Transfer

Skin cancer, the most commonly found human malignancy, is primarily diagnosed visually via dermoscopic analysis, biopsy and histopathological examination. However, unlike other types of cancer, automated image classification of skin lesions is deemed to be more challenging due to the irregularity and variability in the lesions’ appearances. In this work, we propose an adaptation of the Neural Style Transfer (NST) as a novel image pre-processing step for skin lesion classification problems. We represent each dermoscopic image as a style image and transfer the style of the lesion onto a homogeneous content image. This transfers the main variability of each lesion onto the same localized region, which allows us to integrate the generated images together and extract low-rank latent embeddings via tensor decomposition. We evaluated the performance of our model on competition data sets collected and pre-processed from the International Skin Imaging Collaboration (ISIC) database. We show that the classification performance based on the extracted tensor features using the style-transferred images significantly outperforms that of the raw images by more than 10%, and is also competitive with well-studied, pre-trained CNN models using transfer learning. Additionally, the tensor decomposition also affords clinical interpretations and insights by examining the images which correspond to the largest loadings in the top style embedding features as identified by the common supervised learning models.

Cross-Camera Erased Feature Learning for Unsupervised Person Re-Identification

Most supervised person re-identification methods show their excellent performance, but using labeled datasets is very expensive, which limits its application in practical scenarios. To solve the scalability problem, we propose a Cross-camera Erased Feature Learning (CEFL) framework for unsupervised person re-identification that learns discriminative features from image appearances without manual annotations, where both of the cross-camera global image appearance and the local details are explored. Specifically, for the global appearance, in order to bridge the gap between images with the same identities under different cameras, we generate style-transferred images. The network is trained to classify the original images, the style-transferred images and the negative samples. To learn the partial details of the images, we generate erased images and train the network to pull the similar erased images together and push the dissimilar ones away. In addition, we joint learn the discriminative global and local information to learn a more robust model. Global and erased features are used together in feature learning which are successful conjunction of BFENet. A large number of experiments show the superiority of CEFL in unsupervised pedestrian re-identification.

A New Strategy for One-Example Person re-ID: Exploit the Unlabeled Data Gradually Base on Style-Transferred Images

As a research field of symmetry journals, computer vision has received more and more attention. Person re-identification (re-ID) has become a research hotspot in computer vision. We focus on one-example person re-ID, where each person only has one labeled image in the dataset, and other images are unlabeled. There are two main challenges of the task, the insufficient labeled data, and the lack of labeled images cross-cameras. In dealing with the above issue, we propose a new one-example labeling scheme, which generates style-transferred images by CycleGAN (Cycle Generative Adversarial Networks) to ensure that for each person, there is one labeled image under each camera style. Then a self-learning framework is adopted, which iteratively train a CNN (Convolutional Neural Networks) model with labeled images and labeled style-transferred images, and mine the reliable images to assign a pseudo label. The experimental results prove that by integrating the camera style transferred images, we effectively expand the dataset, and the problem of low recognition rate caused by the lack of labeled pedestrian pictures across cameras is effectively solved. Notably, the rank-1 accuracy of our method outperforms the state-of-the-art method by 8.7 points on the Market-1501 dataset, and 6.3 points on the DukeMTMC-ReID dataset.

Unsupervised Person Re-identification via Cross-camera Similarity Exploration.

Most person re-identification (re-ID) approaches are based on supervised learning, which requires manually annotated data. However, it is not only resource-intensive to acquire identity annotation but also impractical for large-scale data. To relieve this problem, we propose a cross-camera unsupervised approach that makes use of unsupervised style-transferred images to jointly optimize a convolutional neural network (CNN) and the relationship among the individual samples for person re-ID. Our algorithm considers two fundamental facts in the re- ID task, i.e., variance across diverse cameras and similarity within the same identity. In this paper, we propose an iterative framework which overcomes the camera variance and achieves across-camera similarity exploration. Specifically, we apply an unsupervised style transfer model to generate style-transferred training images with different camera styles. Then we iteratively exploit the similarity within the same identity from both the original and the style-transferred data. We start with considering each training image as a different class to initialize the Convolutional Neural Network (CNN) model. Then we measure the similarity and gradually group similar samples into one class, which increases similarity within each identity. We also introduce a diversity regularization term in the clustering to balance the cluster distribution. The experimental results demonstrate that our algorithm is not only superior to state-of-the-art unsupervised re-ID approaches, but also performs favorably compared with other competing unsupervised domain adaptation methods (UDA) and semi-supervised learning methods.

Unsupervised Domain Adaptation with Coupled Generative Adversarial Autoencoders

When large-scale annotated data are not available for certain image classification tasks, training a deep convolutional neural network model becomes challenging. Some recent domain adaptation methods try to solve this problem using generative adversarial networks and have achieved promising results. However, these methods are based on a shared latent space assumption and they do not consider the situation when shared high level representations in different domains do not exist or are not ideal as they assumed. To overcome this limitation, we propose a neural network structure called coupled generative adversarial autoencoders (CGAA) that allows a pair of generators to learn the high-level differences between two domains by sharing only part of the high-level layers. Additionally, by introducing a class consistent loss calculated by a stand-alone classifier into the generator optimization, our model is able to generate class invariant style-transferred images suitable for classification tasks in domain adaptation. We apply CGAA to several domain transferred image classification scenarios including several benchmark datasets. Experiment results have shown that our method can achieve state-of-the-art classification results.