Artistic Style Transfer Research Articles

Artistic Style Transfer Based on Attention with Knowledge Distillation

AbstractArtistic style transfer involves the adaption of an input image to reflect the style of a reference image while maintaining its original content. This technique, now a prominent focus due to its prospective use in creative fields like digital art and graphic design, typically applies normalization techniques and attention mechanisms. While these methods yield decent results, they often fall short due to distortion of content image details and non‐artefact styles. In this paper, we introduce a novel approach that synergizes adaptive instance normalization (AdaIN), attention mechanisms, knowledge distillation (KD) and strategically placed internal layers, and new enhancements designed to preserve content details and provide a nuanced control over the style transfer process. We introduce a Detail Enhancement Module to amplify high‐frequency details in the content image, enhancing edge and texture preservation. A Multi‐scale Strategy is implemented to ensure uniform style application across various detail levels, leading to more coherent stylization. The Content Feature Refinement process refines content features, sharpening and emphasizing details to preserve structural and textural integrity. AdaIN's distinctive feature of efficiently collecting style data is exploited in our approach, coupled with attention mechanisms' inherent ability to conserve content information. We supplement this blend with KD for the enhancement of model accuracy and efficiency. Additionally, the introduction of internal layers acts as a conduit to further improve the style transfer process, increasing the transfer level of features and fostering better stylized results. The cornerstone of our technique lies in preserving the content structure amidst complex style transfers. Experimental results affirm the superior performance of our method over existing techniques in both quantitative and qualitative evaluations.

ArtBank: Artistic Style Transfer with Pre-trained Diffusion Model and Implicit Style Prompt Bank

Artistic style transfer aims to repaint the content image with the learned artistic style. Existing artistic style transfer methods can be divided into two categories: small model-based approaches and pre-trained large-scale model-based approaches. Small model-based approaches can preserve the content strucuture, but fail to produce highly realistic stylized images and introduce artifacts and disharmonious patterns; Pre-trained large-scale model-based approaches can generate highly realistic stylized images but struggle with preserving the content structure. To address the above issues, we propose ArtBank, a novel artistic style transfer framework, to generate highly realistic stylized images while preserving the content structure of the content images. Specifically, to sufficiently dig out the knowledge embedded in pre-trained large-scale models, an Implicit Style Prompt Bank (ISPB), a set of trainable parameter matrices, is designed to learn and store knowledge from the collection of artworks and behave as a visual prompt to guide pre-trained large-scale models to generate highly realistic stylized images while preserving content structure. Besides, to accelerate training the above ISPB, we propose a novel Spatial-Statistical-based self-Attention Module (SSAM). The qualitative and quantitative experiments demonstrate the superiority of our proposed method over state-of-the-art artistic style transfer methods. Code is available at https://github.com/Jamie-Cheung/ArtBank.

KBStyle: Fast Style Transfer Using a 200 KB Network with Symmetric Knowledge Distillation.

Convolutional Neural Networks (CNNs) have achieved remarkable progress in arbitrary artistic style transfer. However, the model size of existing state-of-the-art (SOTA) style transfer algorithms is immense, leading to enormous computational costs and memory demand. It makes real-time and high resolution hard for GPUs with limited memory and limits the application on mobile devices. This paper proposes a novel arbitrary artistic style transfer algorithm, KBStyle, whose model size is only 200 KB. Firstly, we design a style transfer network where the style encoder, content encoder, and corresponding decoder are custom designed to guarantee low computational cost and high shape retention. Besides, the weighted style loss function is presented to improve the performance of style migration. Then, we propose a novel knowledge distillation method (Symmetric Knowledge Distillation, SKD) for encoder-decoder-based style transfer models, which redefines the knowledge and symmetrically compresses the encoder and decoder. With the SKD, the proposed style transfer network is further compressed by 14 times to achieve the KBStyle. Experimental results demonstrate that the proposed SKD method achieves comparable results with other SOTA knowledge distillation algorithms for style transfer. Besides, the proposed KBStyle achieves high-quality stylized images. And the inference time of the KBStyle on an Nvidia TITAN RTX GPU is only 20 ms when the resolutions of the content image and style image are both 2k-resolution (2048×1080). Moreover, the 200 KB model size of KBStyle is much smaller than the SOTA models and facilitates style transfer on mobile devices.

Fast continuous patch-based artistic style transfer for videos

Fast continuous patch-based artistic style transfer for videos

Generative AI Model for Artistic Style Transfer Using Convolutional Neural Networks

Artistic style transfer, a captivating application of generative artificial intelligence, involves fusing the content of one image with the artistic style of another to create unique visual compositions. This paper presents a comprehensive overview of a novel technique for style transfer using Convolutional Neural Networks (CNNs). By leveraging deep image representations learned by CNNs, we demonstrate how to separate and manipulate image content and style, enabling the synthesis of high-quality images that combine content and style in a harmonious manner. We describe the methodology, including content and style representations, loss computation, and optimization, and showcase experimental results highlighting the effectiveness and versatility of the approach across different styles and content.

A study of neural artistic style transfer models and architectures for Indian art styles

ABSTRACT Neural Style Transfer (NST) has been a widely researched topic as of late enabling new forms of image manipulation. Here we perform an extensive study on NST algorithms and extend the existing methods with custom modifications for application to Indian art styles. In this paper, we aim to provide a comprehensive analysis of various methods ranging from the seminal work of Gatys et al which demonstrated the power of Convolutional Neural Networks (CNNs) in creating artistic imagery by separating and recombining image content and style, to the state of the art image-to-image translation models which use Generative Adversarial Networks (GANs) to learn the mapping between two domain of images. We observe and infer based on the results produced by the models on which one could be a more suitable approach for Indian art styles, especially Tanjore paintings which are unique compared to the Western art styles. We then propose the method which is more suitable for the domain of Indian Art style along with custom architecture which includes an enhancement and evaluation module. We then present evaluation methods, both qualitative and quantitative which includes our proposed metric, to evaluate the results produced by the model.

Understanding Intuition: Can Rapid Cognition Perform Better than Rational Thinking in Differentiating Artworks between Artist and Artistic Style Transfer

This study attempts to provide evidence that judgements based on rapid cognition can have higher accuracy than judgements based on rational thinking in particular situations. The design of the experiment was based on the previous study by Sun et al. 2022 that compared cognitive differences in artworks between artists and art style transfer. In the experiment of this paper, the stimuli were generated from 24 pairs of digital artworks done by AI and human painters respectively, and participants were asked to differentiate between the stimuli. The results indicated that participants made more correct choices when there was not enough time to process all the details than when there was enough time to consider all the evidence. This study once again demonstrates that rapid cognition holds advantages in analyzing complex information in a short period of time.

CSAST: Content self-supervised and style contrastive learning for arbitrary style transfer

Arbitrary artistic style transfer has achieved great success with deep neural networks, but it is still difficult for existing methods to tackle the dilemma of content preservation and style translation due to the inherent content-and-style conflict. In this paper, we introduce content self-supervised learning and style contrastive learning to arbitrary style transfer for improved content preservation and style translation, respectively. The former one is based on the assumption that stylization of a geometrically transformed image is perceptually similar to applying the same transformation to the stylized result of the original image. This content self-supervised constraint noticeably improves content consistency before and after style translation, and contributes to reducing noises and artifacts as well. Furthermore, it is especially suitable to video style transfer, due to its ability to promote inter-frame continuity, which is of crucial importance to visual stability of video sequences. For the latter one, we construct a contrastive learning that pull close style representations (Gram matrices) of the same style and push away that of different styles. This brings more accurate style translation and more appealing visual effect. A large number of qualitative and quantitative experiments demonstrate superiority of our method in improving arbitrary style transfer quality, both for images and videos.

Deep Deformable Artistic Font Style Transfer

The essence of font style transfer is to move the style features of an image into a font while maintaining the font’s glyph structure. At present, generative adversarial networks based on convolutional neural networks play an important role in font style generation. However, traditional convolutional neural networks that recognize font images suffer from poor adaptability to unknown image changes, weak generalization abilities, and poor texture feature extractions. When the glyph structure is very complex, stylized font images cannot be effectively recognized. In this paper, a deep deformable style transfer network is proposed for artistic font style transfer, which can adjust the degree of font deformation according to the style and realize the multiscale artistic style transfer of text. The new model consists of a sketch module for learning glyph mapping, a glyph module for learning style features, and a transfer module for a fusion of style textures. In the glyph module, the Deform-Resblock encoder is designed to extract glyph features, in which a deformable convolution is introduced and the size of the residual module is changed to achieve a fusion of feature information at different scales, preserve the font structure better, and enhance the controllability of text deformation. Therefore, our network has greater control over text, processes image feature information better, and can produce more exquisite artistic fonts.

Coarse-to-Fine Structure-Aware Artistic Style Transfer

Artistic style transfer aims to use a style image and a content image to synthesize a target image that retains the same artistic expression as the style image while preserving the basic content of the content image. Many recently proposed style transfer methods have a common problem; that is, they simply transfer the texture and color of the style image to the global structure of the content image. As a result, the content image has a local structure that is not similar to the local structure of the style image. In this paper, we present an effective method that can be used to transfer style patterns while fusing the local style structure to the local content structure. In our method, different levels of coarse stylized features are first reconstructed at low resolution using a coarse network, in which style color distribution is roughly transferred, and the content structure is combined with the style structure. Then, the reconstructed features and the content features are adopted to synthesize high-quality structure-aware stylized images with high resolution using a fine network with three structural selective fusion (SSF) modules. The effectiveness of our method is demonstrated through the generation of appealing high-quality stylization results and a comparison with some state-of-the-art style transfer methods.

Comparison of Cognitive Differences of Artworks between Artist and Artistic Style Transfer

This study explores how audiences responded to perceiving and distinguishing the paintings created by AI or human artists. The stimuli were six paintings which were completed by AI and human artists. A total of 750 subjects participated to identify which ones were completed by human artists or by AI. Results revealed that most participants could correctly distinguish between paintings made by AI or human artists and that accuracy was higher for those who used “intuition” as the criterion for judgment. The participants preferred the paintings created by human artists. Furthermore, there were big differences in the perception of the denotation and connotation of paintings between audiences of different backgrounds. The reasons for this will be analyzed in subsequent research.

Digital Image Art Style Transfer Algorithm Based on CycleGAN.

With the continuous development and popularization of artificial intelligence technology in recent years, the field of deep learning has also developed relatively rapidly. The application of deep learning technology has attracted attention in image detection, image recognition, image recoloring, and image artistic style transfer. Some image art style transfer techniques with deep learning as the core are also widely used. This article intends to create an image art style transfer algorithm to quickly realize the image art style transfer based on the generation of confrontation network. The principle of generating a confrontation network is mainly to change the traditional deconvolution operation, by adjusting the image size and then convolving, using the content encoder and style encoder to encode the content and style of the selected image, and by extracting the content and style features. In order to enhance the effect of image artistic style transfer, the image is recognized by using a multi-scale discriminator. The experimental results show that this algorithm is effective and has great application and promotion value.

CartoonLossGAN: Learning Surface and Coloring of Images for Cartoonization.

Cartoonization as a special type of artistic style transfer is a difficult image processing task. The current existing artistic style transfer methods cannot generate satisfactory cartoon-style images due to that artistic style images often have delicate strokes and rich hierarchical color changes while cartoon-style images have smooth surfaces without obvious color changes, and sharp edges. To this end, we propose a cartoon loss based generative adversarial network (CartoonLossGAN) for cartoonization. Particularly, we first reuse the encoder part of the discriminator to build a compact generative adversarial network (GAN) based cartoonization architecture. Then we propose a novel cartoon loss function for the architecture. It can imitate the process of sketching to learn the smooth surface of the cartoon image, and imitate the coloring process to learn the coloring of the cartoon image. Furthermore, we also propose an initialization strategy, which is used in the scenario of reusing the discriminator to make our model training easier and more stable. Extensive experimental results demonstrate that our proposed CartoonLossGAN can generate fantastic cartoon-style images, and outperforms four representative methods.

Interactive Artistic Multi-style Transfer

Artistic style transfer is to render an image in the style of another image, which is a challenge problem in both image processing and arts. Deep neural networks are adopted to artistic style transfer and achieve remarkable success, such as AdaIN (adaptive instance normalization), WCT (whitening and coloring transforms), MST (multimodal style transfer), and SEMST (structure-emphasized multimodal style transfer). These algorithms modify the content image as a whole using only one style and one algorithm, which is easy to cause the foreground and background to be blurred together. In this paper, an iterative artistic multi-style transfer system is built to edit the image with multiple styles by flexible user interaction. First, a subjective evaluation experiment with art professionals is conducted to build an open evaluation framework for style transfer, including the universal evaluation questions and personalized answers for ten typical artistic styles. Then, we propose the interactive artistic multi-style transfer system, in which an interactive image crop tool is designed to cut a content image into several parts. For each part, users select a style image and an algorithm from AdaIN, WCT, MST, and SEMST by referring to the characteristics of styles and algorithms summarized by the evaluation experiments. To obtain richer results, the system provides a semantic-based parameter adjustment mode and the function of preserving colors of content image. Finally, case studies show the effectiveness and flexibility of the system.

PortraitNET: Photo-realistic portrait cartoon style transfer with self-supervised semantic supervision

We propose a novel framework to transfer the portrait image into its correspondence with photo-realistic and cartoon style. The existing work on neural style transfer conducts impressive results on artistic style transfer; however, the lack of semantic clues will lead to the color artifacts in photo-realistic style transfer because of the complex background and noise issues. In this work, we re-define the semantics as the pixel motion field according to the color displacement between adjacent animation frames along the optical direction and initiatively propose the self-supervised semantic network (SSNet) to learn semantic maps without human inference or any priories. The SSNet shares parameters with the style transfer network; thus, the superior alternatives can preserve the semantic completeness in the styled image. To solve the content missing and blur problems common in NST, we propose the bilateral convolution block (B-block) and feature fusion strategy (F-block) for visual smoothness to meet the perceptive satisfaction. The ablation studies are provided to validate the effectiveness, and comparative experiments with the state-of-the-art baselines demonstrate the advantages of the proposed method.

Interpretable Actions: Controlling Experts with Understandable Commands

Despite the prevalence of deep neural networks, their single most cited drawback is that, even when successful, their operations are inscrutable. For many applications, the desired outputs are the composition of externally-defined bases. For such decomposable domains, we present a two-stage learning procedure producing combinations of the external bases which are trivially extractable from the network. In the first stage, the set of external bases that will form the solution are modeled as differentiable generator modules, controlled by the same parameters as the external bases. In the second stage, a controller network is created that selects parameters for those generators, either successively or in parallel, to compose the final solution. Through three tasks, we concretely demonstrate how our system yields readily understandable commands. In one, we introduce a new form of artistic style transfer, learning to draw and color with crayons, in which the transformation of a photograph or painting occurs not as a single monolithic computation, but by the composition of thousands of individual, visualizable strokes. The other two tasks, single-pass function approximation with arbitrary bases and shape-based synthesis, show how our approach produces understandable and extractable actions in two disparate domains.

Comparing optimization methods for deep learning in image processing applications

Abstract Training a deep learning network requires choosing its weights such that the output minimizes a given loss function. In practice, stochastic gradient descent is frequently used for solving the optimization problem. Several variants of this approach have been suggested in the literature. We study the impact of the choice of the optimization method on the outcome of the learning process at the example of two image processing applications from quite different fields. The first one is artistic style transfer, where the content of one image is combined with the style of another one. The second application is a real world classification task from industry, namely detecting defects in images of air filters. In both cases, clear differences between the results of the individual optimization methods are observed.

The Cognition of Audience to Artistic Style Transfer

Artificial Intelligence (AI) is becoming more popular in various fields, including the area of art creation. Advances in AI technology bring new opportunities and challenges in the creation, experience, and appreciation of art. The neural style transfer (NST) realizes the intelligent conversion of any artistic style using neural networks. However, the artistic style is the product of cognition that involving from visual to feel. The purpose of this paper is to study factors affecting audience cognitive difference and preference on artistic style transfer. Those factors are discussed to investigate the application of the AI generator model in art creation. Therefore, based on the artist’s encoding attributes (color, stroke, texture) and the audience’s decoding cognitive levels (technical, semantic, effectiveness), this study proposed a framework to evaluate artistic style transfer in the perspective of cognition. Thirty-one subjects with a background in art, aesthetics, and design were recruited to participate in the experiment. The experimental process consists of four style groups, including Fauvism, Expressionism, Cubism, and Renaissance. According to the finding in this study, participants can still recognize different artistic styles after transferred by neural networks. Besides, the features of texture and stroke are more impact on the perception of fitness than color. The audience may prefer the samples with high cognition in the semantic and effectiveness levels. The above indicates that through AI automated routine work, the cognition of the audience to artistic style still can be kept and transferred.

Comparison of Cognitive Differences of Artworks between Artist and Artistic Style Transfer

Comparison of Cognitive Differences of Artworks between Artist and Artistic Style Transfer

The artistic style transfer from Shanghai modern landmark buildings images to Xiao Jiaochang New Year pictures based on deep learning

Posting New Year pictures is a folk custom with a long history in China. Shanghai New Year pictures are loved by the people with its unique Shanghai style. As a representative of Shanghai New Year pictures, Xiao Jiaochang New Year pictures have left a strong mark in the history of Shanghai style culture and history through its diverse and comprehensive categories and rich artistic styles. However, due to its short well-developed period, the New Year pictures in Xiao Jiaochang have no inheritance and cannot apply for intangible cultural heritage. Nowadays, it is not realistic for us to use the traditional way to make New Year pictures in small schools because of the lack of inheritors and the production techniques have not been continued. Therefore, this paper mainly focuses on tracing back to this period of history, and realize the recording and dissemination of the style of the New Year pictures of Xiao Jiaochang through modern high-tech. Collect the relatively complete New Year pictures of Xiao Jiaochang in the world as a data set, and realize the style transfer to modern Shanghai’s landmark buildings’ pictures through the method of machine learning. Realizing the electronic work on spreading the culture in addition to the normal electronic record needs to be in time, so that Xiao Jiaochang New Year pictures can reappear in the world.