Image Style Transfer Research Articles

Hairstyle-and-identity-aware facial image style transfer with region-guiding masks

Hairstyle-and-identity-aware facial image style transfer with region-guiding masks

Large capacity generative image steganography via image style transfer and feature-wise deep fusion

Large capacity generative image steganography via image style transfer and feature-wise deep fusion

Dual-Affinity Style Embedding Network for Semantic-Aligned Image Style Transfer.

Image style transfer aims at synthesizing an image with the content from one image and the style from another. User studies have revealed that the semantic correspondence between style and content greatly affects subjective perception of style transfer results. While current studies have made great progress in improving the visual quality of stylized images, most methods directly transfer global style statistics without considering semantic alignment. Current semantic style transfer approaches still work in an iterative optimization fashion, which is impractically computationally expensive. Addressing these issues, we introduce a novel dual-affinity style embedding network (DaseNet) to synthesize images with style aligned at semantic region granularity. In the dual-affinity module, feature correlation and semantic correspondence between content and style images are modeled jointly for embedding local style patterns according to semantic distribution. Furthermore, the semantic-weighted style loss and the region-consistency loss are introduced to ensure semantic alignment and content preservation. With the end-to-end network architecture, DaseNet can well balance visual quality and inference efficiency for semantic style transfer. Experimental results on different scene categories have demonstrated the effectiveness of the proposed method.

Research advanced in image style transfer based on deep learning

Image style transfer aims to change a picture’s style while maintaining the integrity of the image’s original information. Image style transfer techniques based on deep learning have expanded in variety as a result of the quick advancement of artificial intelligence technology, and have made breakthroughs in both transfer accuracy and visualization effect. Focusing on image style transfer based on depth learning, this paper systematically introduces its latest research progress. Specifically, according to the difference of network structure, this paper introduces four representative style transfer methods based on CNN and GAN, including their design ideas, process based, advantages and disadvantages. In addition, this paper also qualitatively and quantitatively compares the transfer results of different algorithms on different datasets. For GAN based method, AnimeGAN has a good conversion effect in style conversion. For the CNN based method, the combination of perceptual loss and feedforward network using perceptual loss is more prominent. Finally, this paper reviews the development and breakthrough of neural network style, and sorts out and discusses the differences and shortcomings of these technologies as well as some unresolved problems.

WCGAN: Robust portrait watercolorization with adaptive hierarchical localized constraints

Deep learning has enabled image style transfer to make great strides forward. However, unlike many other styles, transferring the watercolor style to portraits is significantly challenging in image synthesis and style transfer. Pixel-correlation-based methods do not produce satisfactory watercolors. This is because portrait watercolors exhibit the sophisticated fusion of various painting techniques in local areas, which poses a problem for convolutional neural networks to accurately handle fine-grained features. Moreover, the common but problematic way of coping with multiple scales greatly impedes the performance of existing style transfer methods with fixed receptive fields. Although it is possible to develop an image processing pipeline mimicking various watercolor effects, such algorithms are slow and fragile, especially for inputs of different scales. As a remedy, this paper proposes WCGAN, a generative adversarial network (GAN) architecture dedicated to watercolorization of portraits. Specifically, a novel localized style loss suitable for watercolorization is proposed to deal with local details. To handle portraits of different scales and improve robustness, a novel discriminator architecture with three parallel branches of varying sizes of receptive fields is introduced. In addition, the application of WCGAN is expanded to video style transfer where a novel kind of video training data based on random crops is developed to efficiently capture temporal consistency. Extensive experimental results from qualitative and quantitative analyses demonstrate that WCGAN generates state-of-the-art, high quality watercolors from portraits.

Unpaired Artistic Portrait Style Transfer via Asymmetric Double-Stream GAN.

With the development of image style transfer technologies, portrait style transfer has attracted growing attention in this research community. In this article, we present an asymmetric double-stream generative adversarial network (ADS-GAN) to solve the problems that caused by cartoonization and other style transfer techniques when they are applied to portrait photos, such as facial deformation, contours missing, and stiff lines. By observing the characteristics between source and target images, we propose an edge contour retention (ECR) regularized loss to constrain the local and global contours of generated portrait images to avoid the portrait deformation. In addition, a content-style feature fusion module is introduced for further learning of the target image style, which uses a style attention mechanism to integrate features and embeds style features into content features of portrait photos according to the attention weights. Finally, a guided filter is introduced in content encoder to smooth the textures and specific details of source image, thereby eliminating its negative impact on style transfer. We conducted overall unified optimization training on all components and got an ADS-GAN for unpaired artistic portrait style transfer. Qualitative comparisons and quantitative analyses demonstrate that the proposed method generates superior results than benchmark work in preserving the overall structure and contours of portrait; ablation and parameter study demonstrate the effectiveness of each component in our framework.

A Unified Arbitrary Style Transfer Framework via Adaptive Contrastive Learning

This work presents Unified Contrastive Arbitrary Style Transfer (UCAST), a novel style representation learning and transfer framework, that can fit in most existing arbitrary image style transfer models, such as CNN-based, ViT-based, and flow-based methods. As the key component in image style transfer tasks, a suitable style representation is essential to achieve satisfactory results. Existing approaches based on deep neural networks typically use second-order statistics to generate the output. However, these hand-crafted features computed from a single image cannot leverage style information sufficiently, which leads to artifacts such as local distortions and style inconsistency. To address these issues, we learn style representation directly from a large number of images based on contrastive learning by considering the relationships between specific styles and the holistic style distribution. Specifically, we present an adaptive contrastive learning scheme for style transfer by introducing an input-dependent temperature. Our framework consists of three key components: a parallel contrastive learning scheme for style representation and transfer, a domain enhancement (DE) module for effective learning of style distribution, and a generative network for style transfer. Qualitative and quantitative evaluations show the results of our approach are superior to those obtained via state-of-the-art methods. The code is available at https://github.com/zyxElsa/CAST_pytorch .

The Generation of Articulatory Animations Based on Keypoint Detection and Motion Transfer Combined with Image Style Transfer

Knowing the correct positioning of the tongue and mouth for pronunciation is crucial for learning English pronunciation correctly. Articulatory animation is an effective way to address the above task and helpful to English learners. However, articulatory animations are all traditionally hand-drawn. Different situations require varying animation styles, so a comprehensive redraw of all the articulatory animations is necessary. To address this issue, we developed a method for the automatic generation of articulatory animations using a deep learning system. Our method leverages an automatic keypoint-based detection network, a motion transfer network, and a style transfer network to generate a series of articulatory animations that adhere to the desired style. By inputting a target-style articulation image, our system is capable of producing animations with the desired characteristics. We created a dataset of articulation images and animations from public sources, including the International Phonetic Association (IPA), to establish our articulation image animation dataset. We performed preprocessing on the articulation images by segmenting them into distinct areas each corresponding to a specific articulatory part, such as the tongue, upper jaw, lower jaw, soft palate, and vocal cords. We trained a deep neural network model capable of automatically detecting the keypoints in typical articulation images. Also, we trained a generative adversarial network (GAN) model that can generate end-to-end animation of different styles automatically from the characteristics of keypoints and the learned image style. To train a relatively robust model, we used four different style videos: one magnetic resonance imaging (MRI) articulatory video and three hand-drawn videos. For further applications, we combined the consonant and vowel animations together to generate a syllable animation and the animation of a word consisting of many syllables. Experiments show that this system can auto-generate articulatory animations according to input phonetic symbols and should be helpful to people for English articulation correction.

Synthetic-real image domain adaptation for indoor camera pose regression using a 3D model

Deep learning-based camera pose regression approaches have achieved outstanding performance for visual indoor localisation. However, these approaches are limited by the availability of images with known camera poses, and they often require a comprehensive mapping of the indoor scenes, which is labour-intensive and often impractical. Recent studies have shown that synthetic images derived from simple 3D building models can be used to train deep learning models to perform cross-domain synthetic-to-real visual localisation. But the performance of such cross-domain localisation models is degraded due to the domain gap between the real and synthetic images. In this study, we propose a domain adaptation approach based on a Generative Adversarial Network (GAN) framework, which is trained on a set of unpaired synthetic and real images to generate real-looking synthetic images and synthetic-looking real images. These images are then used to train the current deep learning-based camera pose regression networks. We develop two domain adaptation strategies for the localisation task, namely, synthetic-to-real and real-to-synthetic image style transfer. The results show that the best model is obtained for the real-to-synthetic style transfer with a localisation error of 0.31 m and orientation error of 3.36°, which is approximately three times lower compared to the previously reported results.

Artistic Image Style Transfer Based on CycleGAN Network Model

Artistic Image Style Transfer Based on CycleGAN Network Model

Speckle noise reduction for medical ultrasound images based on cycle-consistent generative adversarial network

Medical ultrasound (US) images are corrupted by speckle noise, which can adversely affect the disease diagnosis and treatment. Recently, the cycle-consistent adversarial network (CycleGAN) has been used in the style transfer for both natural and medical images. In this study, we aim to develop a US despeckling method based on the CycleGAN by the style transfer between the speckled noisy data domain and noise-free data domain by forming a bi-directional universal mapping. The inputs of noisy and noise-free images are designed in the CycleGAN model. For simulation work, we use both the multiplicative model and the spatial impulse response model to obtain noisy images from noise-free images. However, noise-free US images are not clinically available. Hence, for the real US despeckling scenario, the clinical images of hearts, lymph nodes, and breast tumors are used as noisy images; and the high-quality images that are derived from the clinical images by despeckling with the Gabor-based anisotropic diffusion (GAD) and selected with a new metric named the edge-to-noise ratio, are used as the noise-free images. We compare our CycleGAN based denoising method with nine existing denoising methods, i.e., the speckle reduction anisotropic diffusion, GAD, non-local means, wavelet transform, unbiased non-local means, statistical nearest-neighbors, TVHTVM, improved non-local self-similarity measures, and generative adversarial network. Our method outperforms other methods by visual assessment and quantitative comparison, which demonstrates its superiority for noise reduction and detail preservation.

An image style transfer algorithm based on attention mechanism and GhostNet

Image style transfer, which combines the content of one image with the style of another to create a new image, has many potential applications in the disciplines of image creation, creative style reproduction, animation production, and other areas. The majority of early image style transfer techniques used manual features. Due to advances in deep learning technology, the Generative Adversarial Network (GAN) method greatly increases the accuracy of image style transfer; however, when it comes to scenes with large color blocks, high resolution, or horizontal boundaries, their performance still falls short of practical application requirements. In this research, we present a GhostNet- and attention-based image style transfer technique. We specifically introduce the SELayer to learn and increase the weight value of each channel, which can improve the quality of style transfer and data processing capacity. In addition, GhostNet is included to accelerate image production even more. The findings of the experiments, both quantitative and qualitative, demonstrate that the suggested technique can enhance the impact of style transfer.

Research advanced in image style transfer based on deep learning

Image style transfer (IST) aims to combine the style of one image with the content of another to make the resulting image has both the attributes of the style image and the characteristics of the content image. Benefited from rapid developments of deep learning, the methods based on Generative Adversarial Networks (GAN) have greatly improved the accuracy of IST. In this paper, to promote the research of IST based on Convolutional Neural Networks (CNN), some main methods of it are presented. The developments of CNN and GAN are reviewed, and basic principles and techniques of IST based on CNN and GAN are introduced in detail. Moreover, some representative datasets for this task are summarized, then the evaluation metrics and results are presented. Moreover, some problems of current methods are discussed. Finally, application prospects in related fields and future research directions of IST are presented.

Multi-scaled attentive style transfer based on dilated convolution

Image style transfer aims to apply artists painting styles to various images. Many approaches seek different purposes, but a general tendency is to increase efficiency and enable arbitrary style inputs. The state-of-art method is the adaptive convolutional network which expands the process of feature mixture into a layer-wise adjustment that superior previous work by presenting results that is more aware of detailed structures. However, the encoding process that guides the entire stylized revision is unaware of multi-scaled information. We designed an improved version of the style feature encoding procedure in our work. With the introduction of dilated convolution with a different rate, the output stylized image is better at spatial texture migration and color distribution determination. We also come up with a hybrid objective that better measures the spatial dissimilarity between content and style features.

CSAST: Content self-supervised and style contrastive learning for arbitrary style transfer

Arbitrary artistic style transfer has achieved great success with deep neural networks, but it is still difficult for existing methods to tackle the dilemma of content preservation and style translation due to the inherent content-and-style conflict. In this paper, we introduce content self-supervised learning and style contrastive learning to arbitrary style transfer for improved content preservation and style translation, respectively. The former one is based on the assumption that stylization of a geometrically transformed image is perceptually similar to applying the same transformation to the stylized result of the original image. This content self-supervised constraint noticeably improves content consistency before and after style translation, and contributes to reducing noises and artifacts as well. Furthermore, it is especially suitable to video style transfer, due to its ability to promote inter-frame continuity, which is of crucial importance to visual stability of video sequences. For the latter one, we construct a contrastive learning that pull close style representations (Gram matrices) of the same style and push away that of different styles. This brings more accurate style translation and more appealing visual effect. A large number of qualitative and quantitative experiments demonstrate superiority of our method in improving arbitrary style transfer quality, both for images and videos.

An improved defocusing adaptive style transfer method based on a stroke pyramid.

Image style transfer aims to assign a specified artist's style to a real image. However, most existing methods cannot generate textures of various thicknesses due to the rich semantic information of the input image. The image loses some semantic information through style transfer with a uniform stroke size. To address the above problems, we propose an improved multi-stroke defocus adaptive style transfer framework based on a stroke pyramid, which mainly fuses various stroke sizes in the image spatial dimension to enhance the image content interpretability. We expand the receptive field of each branch and then fuse the features generated by the multiple branches based on defocus degree. Finally, we add an additional loss term to enhance the structural features of the generated image. The proposed model is trained using the Common Objects in Context (COCO) and Synthetic Depth of Field (SYNDOF) datasets, and the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) are used to evaluate the overall quality of the output image and its structural similarity with the content image, respectively. To validate the feasibility of the proposed algorithm, we compare the average PSNR and SSIM values of the output of the modified model and those of the original model. The experimental results show that the modified model improves the PSNR and SSIM values of the outputs by 1.43 and 0.12 on average, respectively. Compared with the single-stroke style transfer method, the framework proposed in this study improves the readability of the output images with more abundant visual expression.

Aesthetic style transferring method based on deep neural network between Chinese landscape painting and classical private garden’s virtual scenario

ABSTRACT Most of the existing virtual scenarios built for the digital protection of Chinese classical private gardens are too modern in expression style to show the aesthetic significance of their historical period. Considering the aesthetic commonality between traditional Chinese landscape paintings and classical private gardens and referring to image style transfer, here, a deep neural network was proposed to transfer the aesthetic style from landscape paintings to the virtual scenario of classical private gardens. The network consisted of two parts: style prediction and style transfer. The style prediction network was used to obtain style representation from style paintings, and the style transfer network was used to transfer style representation to the content scenario. The pre-trained network was then embedded into the scenario rendering pipeline and combined with the screen post-processing method to realise the stylised expression of the virtual scenario. To verify the feasibility of this methodology, a virtual scenario of the Humble Administrator’s Garden was used as the content scenario and five garden landscape paintings from different time periods and painting styles were selected for the case study. The results demonstrated that this methodology could effectively achieve the aesthetic style transfer of a virtual scenario.

Image Style Transfer Based on Dynamic Convolutional Manifold Alignment of Halo Attention

The objective of image style transfer is to render an image with artistic features of a style reference while preserving the details of the content image. With the development of deep learning, many arbitrary style transfer methods have emerged. From the recent arbitrary style transfer algorithms, it has been found that the images generated suffer from the problem of poorly stylized quality. To solve this problem, we propose an arbitrary style transfer algorithm based on halo attention dynamic convolutional manifold alignment. First, the features of the content image and style image are extracted by a pre-trained VGG encoder. Then, the features are extracted by halo attention and dynamic convolution, and then the content feature space and style feature space are aligned by attention operations and spatial perception interpolation. The output is achieved through dynamic convolution and halo attention. During this process, multi-level loss functions are used, and total variation loss is introduced to eliminate noise. The manifold alignment process is then repeated three times. Finally, the pre-trained VGG decoder is used to output the stylized image. The experimental results show that our proposed method can generate high-quality stylized images, achieving values of 33.861, 2.516, and 3.602 for ArtFID, style loss, and content loss, respectively. A qualitative comparison with existing algorithms showed that it achieved good results. In future work, we will aim to make the model lightweight.

Artificial Intelligence for Art Creation with Image Style

Artificial intelligence (AI) has become a great success in the past decade. Powered by efficient hardware and neural network models, AI has made a wide change to almost all disciplines. While still in an early stage, AI has been used in the art field much more frequently than before, with various methods, e.g., generative adversarial networks. However, whether AI can replace human beings in art creation is still an unsolved question. Therefore, how to use AI technology to achieve artistic creation has become very meaningful, and has also been favored by many research groups. In this paper, we demonstrate the ability of AI for art creation with a case study of image style transfer. The result shows that AI is still not satisfactory and consumes too much computation. However, a new creative idea is proposed, and we will conduct lightweight research on this basis in the future. So as to realize the artistic creation of real artificial intelligence.

Studies Advanced in Image Style Transfer based on Deep Learning

Image style transfer (IST) is a hot topic in the computer vision community, which refers to learning the distribution of a given style image to convert any image into corresponding image style while the content of the original image is preserved as much as possible. Early style transfer mainly utilizes texture features. Thanks to the great improvement of deep learning technology, researches on IST based on convolutional neural networks (CNN) have achieved breakthroughs in accuracy and speed. Focusing on the topic of deep learning-based IST, we will introduce the latest algorithms in detail, including their basic ideas, key steps, advantages, and disadvantages. Also, we will give an analysis of the performance of representative methods. Furthermore, we discuss the problems to be solved in style transfer and summarize the challenges and development trends in the future.