Style Transfer Network Research Articles

Photorealistic attention style transfer network for architectural photography photos

Architectural photography style transfer, a task in computer vision, employs deep learning algorithms to transform the style of architectural photograph while preserving key structure and content. Existing algorithms face challenges due to the intricate details of buildings, including diverse shapes, lines, and textures. Moreover, considerations for artistic effects in architectural photography style transfer, such as lighting, shadows, and atmosphere, require high-quality image generation algorithms. However, current algorithms often struggle to address these complexities, leading to loss or blurring of details and less realistic images. To overcome these challenges, this paper proposes a Photorealistic Attention Style Transfer Network. The proposed approach utilizes a semantic segmentation model to accurately segment the input image into foreground and background components for independent style transfer. Subsequently, the transferred images are refined by focusing on intricate building parts using the coordinate attention mechanism. Additionally, the network incorporates loss functions to capture light, shadow, and colors in stylish images, ensuring realism while maintaining aesthetic appeal. Through comparative experiments, the proposed network shows better performance in terms of image fidelity and overall aesthetics, and the SSIM and PSNR indices are also better than the current mainstream methods.

Few-Shot Face Sketch-to-Photo Synthesis via Global-Local Asymmetric Image-to-Image Translation

Face sketch-to-photo synthesis is widely used in law enforcement and digital entertainment, which can be achieved by Image-to-Image (I2I) translation. Traditional I2I translation algorithms usually regard the bidirectional translation of two image domains as two symmetric processes, so the two translation networks adopt the same structure. However, due to the scarcity of face sketches and the abundance of face photos, the sketch-to-photo and photo-to-sketch processes are asymmetric. Considering this issue, we propose a few-shot face sketch-to-photo synthesis model based on asymmetric I2I translation, where the sketch-to-photo process uses a feature-embedded generating network, while the photo-to-sketch process uses a style transfer network. On this basis, a three-stage asymmetric training strategy with style transfer as the trigger is proposed to optimize the proposed model by utilizing the advantage that the style transfer network only needs few-shot face sketches for training. Additionally, we discover that stylistic differences between the global and local sketch faces lead to inconsistencies between the global and local sketch-to-photo processes. Thus, a dual branch of the global face and local face is adopted in the sketch-to-photo synthesis model to learn the specific transformation processes for global structure and local details. Finally, the high-quality synthetic face photo can be generated through the global-local face fusion sub-network. Extensive experimental results demonstrate that the proposed Global-Local Asymmetric (GLAS) I2I translation algorithm compared to SOTA methods, at least improves FSIM by 0.0126, and reduces LPIPS (alex), LPIPS (squeeze), and LPIPS (vgg) by 0.0610, 0.0883, and 0.0719, respectively.

Deep learning based method for dynamic tracking of bubbles in microfluidic gas-driven water experiments

Using microfluidic chips for gas displacement experiments is an important approach for studying CO2 displacement mechanisms and gas-liquid exchange processes. Analyzing the micro-scale distribution of gas and water during gas displacement enables the optimization of injection strategies, offering valuable guidance for enhancing oil and gas recovery rates. Traditional image processing techniques often face challenges in effectively handling the complex gas-liquid flow dynamics and irregular bubble shapes encountered in gas displacement experiments. To address this challenge, this paper proposes a gas bubble detection and tracking method for gas-driven water microfluidic experiments based on domain adaptation in deep learning and improved YOLOv8. Using the CycleGAN style transfer network to generate numerous simulated bubble training samples significantly reduces the manual labeling workload for bubbles. Additionally, the SE attention mechanism is integrated into the YOLOv8 backbone network to enhance the feature extraction capability of bubbles, and the Fast-PConv Head structure is incorporated into the detection head to enhance detection efficiency. Experimental results demonstrate that using just 20 annotated experimental images, the bubble detection accuracy of the proposed method can reach 94%. Lastly, DeepSORT is employed to track bubbles and visualize the dynamic trends of bubble changes in the form of visual charts. The deep learning approach proposed in this paper provides a new perspective for intelligent analysis of bubbles in microfluidic chip experiments, and future work aiming to its application to the complex multiphase flow field in porous media.

Arbitrary style transfer via multi-feature correlation

Recent research in arbitrary style transfer has highlighted challenges in maintaining the balance between content structure and style patterns. Moreover, the improper application of style patterns onto the content image often results in suboptimal quality. In this paper, a novel style transfer network, called MCNet, is proposed. It is based on multi-feature correlations. To better explore the intrinsic relationship between the style image and the content image and to transfer the most suitable style onto the content image, a novel Global Style-Attentional Transfer Module, named GSATM, is introduced in this work. GSATM comprises two parts: Forward Adaptive Style Transformation (FAST) and Delayed Style Transformation (DST). The former analyzes the relationship between style and content features and fine-tunes the style features, whereas the latter transfers the content features based on the fine-tuned style features. Moreover, a new encoding and decoding structure is designed to effectively handle the output of GSATM. Extensive quantitative and qualitative experiments fully demonstrate the superiority of our algorithm. Project page: https://github.com/XiangJinCherry/MCNet.

FISTNet: FusIon of STyle-path generative Networks for facial style transfer

With the surge in emerging technologies such as Metaverse, spatial computing, and generative AI, the application of facial style transfer has gained much interest from researchers and startups enthusiasts alike. StyleGAN methods have paved the way for transfer-learning strategies that could reduce the dependency on the vast data available for the training process. However, StyleGAN methods tend to need to be more balanced, resulting in the introduction of artifacts in the facial images. Studies such as DualStyleGAN proposed multipath networks but required the networks to be trained for a specific style rather than simultaneously generating a fusion of facial styles. In this paper, we propose a Fusion of STyles (FIST) network for facial images that leverages pretrained multipath style transfer networks to eliminate the problem associated with the lack of enormous data volume in the training phase and the fusion of multiple styles at the output. We leverage pretrained styleGAN networks with an external style pass that uses a residual modulation block instead of a transform coding block. The method also preserves facial structure, identity, and details via the gated mapping unit introduced in this study. The aforementioned components enable us to train the network with minimal data while generating high-quality stylized images, opening up new possibilities for facial style transfer in emerging technologies. Our training process adapts curriculum learning strategy to perform efficient, flexible style, and model fusion in the generative space. We perform extensive experiments to show the superiority of the proposed FISTNet compared to existing state-of-the-art methods.

Multimodal Content Analysis Using Deep Learning

Abstract: The multimodal content analysis platform combines sentiment analysis and neural style transfertechniques to process and improve various types of digital content. The sentiment analysis module utilizes natural language processing (NLP) algorithms, such as recurrent neural networks (RNNs) or transformer models like BERT, to extract emotional signals from textual, visual, and auditory inputs. Signals are classified into predefined sentiment categories, providing granular insights into the emotional context of the content. The platform employs neural style transfer algorithms, such as style transfer networks (NSTNs) or generative adversarial networks (GANs), to transfer stylistic attributes between texts. By training on a diverse range of artistic styles, the system learns to apply these styles to input text while preserving semantic meaning. This process enhances the visual representation of textual content, making it more appealing and engaging to users.

TIST-Net: style transfer in dynamic contrast enhanced MRI using spatial and temporal information

Objective. Training deep learning models for image registration or segmentation of dynamic contrast enhanced (DCE) MRI data is challenging. This is mainly due to the wide variations in contrast enhancement within and between patients. To train a model effectively, a large dataset is needed, but acquiring it is expensive and time consuming. Instead, style transfer can be used to generate new images from existing images. In this study, our objective is to develop a style transfer method that incorporates spatio-temporal information to either add or remove contrast enhancement from an existing image. Approach. We propose a temporal image-to-image style transfer network (TIST-Net), consisting of an auto-encoder combined with convolutional long short-term memory networks. This enables disentanglement of the content and style latent spaces of the time series data, using spatio-temporal information to learn and predict key structures. To generate new images, we use deformable and adaptive convolutions which allow fine grained control over the combination of the content and style latent spaces. We evaluate our method, using popular metrics and a previously proposed contrast weighted structural similarity index measure. We also perform a clinical evaluation, where experts are asked to rank images generated by multiple methods. Main Results. Our model achieves state-of-the-art performance on three datasets (kidney, prostate and uterus) achieving an SSIM of 0.91 ± 0.03, 0.73 ± 0.04, 0.88 ± 0.04 respectively when performing style transfer between a non-enhanced image and a contrast-enhanced image. Similarly, SSIM results for style transfer from a contrast-enhanced image to a non-enhanced image were 0.89 ± 0.03, 0.82 ± 0.03, 0.87 ± 0.03. In the clinical evaluation, our method was ranked consistently higher than other approaches. Significance. TIST-Net can be used to generate new DCE-MRI data from existing images. In future, this may improve models for tasks such as image registration or segmentation by allowing small training datasets to be expanded.

Extending user control for image stylization using hierarchical style transfer networks

The field of neural style transfer refers to the re-rendering of content image while fusing the features of a style image. The recent studies either focus on multiple style transfer or arbitrary style transfer while using perceptual and fixpoint content losses in their respective network architectures. The aforementioned losses provide notable stylization results but lack the liberty of style control to the user. Consequently, the stylization results also compromise the preservation of details with respect to the content image. This work proposes the hierarchical style transfer network (HSTN) for the image stylization task that could provide the user with the liberty to control the degree of incurred style via denoising parameter. The HSTN incorporates the proposed fixpoint control loss that preserves details from the content image and the addition of denoising CNN network (DnCNN) and denoising loss for allowing the user to control the level of stylization. The encoder-decoder block, the DnCNN block, and the loss network block make the basic building blocks of HSTN. Extensive experiments have been carried out, and the results are compared with existing works to demonstrate the effectiveness of HSTN. The subjective user evaluation shows that the HSTN's stylization represents the best fusion of style and generates unique stylization results while preserving the content image details, which is evident by acquiring 12% better results than the second-best performing method. It has also been observed that the proposed work is amongst the studies that achieve the best trade-off regarding content and style classification scores, i.e. 37.64% and 60.27%, respectively.

Enhancing image style transfer for real-time indoor geometric data using GAN

<p>The need for two-dimensional (2D) simulated environments, such as those seen in VR/AR and the multiverse, has grown substantially in the rapidly emerging modern era. To reduce the need for human interaction, the present studies have focused on the automated transfer of image processing styles inside a 2D virtual world through the application of computer vision. However, there are several limits to the current research on 2D environment-style transfers that rely on modelling. One thing to keep in mind is that there is a significant amount of style image data required for training a style transfer network specifically for 2D simulations. All this information must be combined with perspectives that are fairly accurate representations of the internet. The second issue was that the 2D structures were inconsistent. Most of the research relies on 2D input image attributes and ignores 2D scene geometric data. Lastly, changing the way something looks does not change the fact that every object has its own distinct qualities. To address these issues, we propose an enhanced Generative Adversarial Networks (GAN) approach for image style transfer in which oversampling and fine tuning is used to improve the data loss. Image Style transfer is an approach to image-to-image translation that supports the transfer of the style of an image to a real-time image and is mainly used to enhance the resolution and quality of an image. The performance of the proposed approach is analyzed using the MIT dataset, which is retrieved from Kaggle. It is an open-source dataset that contains 67 indoor categories and has more than 15,000 images. The proposed approach is implemented using MATLAB simulation tool. The results of proposed approach show that, content loss and style loss are lower than other deep learning models such as VGG16 and Alex Net.</p>

KBStyle: Fast Style Transfer Using a 200 KB Network with Symmetric Knowledge Distillation.

Convolutional Neural Networks (CNNs) have achieved remarkable progress in arbitrary artistic style transfer. However, the model size of existing state-of-the-art (SOTA) style transfer algorithms is immense, leading to enormous computational costs and memory demand. It makes real-time and high resolution hard for GPUs with limited memory and limits the application on mobile devices. This paper proposes a novel arbitrary artistic style transfer algorithm, KBStyle, whose model size is only 200 KB. Firstly, we design a style transfer network where the style encoder, content encoder, and corresponding decoder are custom designed to guarantee low computational cost and high shape retention. Besides, the weighted style loss function is presented to improve the performance of style migration. Then, we propose a novel knowledge distillation method (Symmetric Knowledge Distillation, SKD) for encoder-decoder-based style transfer models, which redefines the knowledge and symmetrically compresses the encoder and decoder. With the SKD, the proposed style transfer network is further compressed by 14 times to achieve the KBStyle. Experimental results demonstrate that the proposed SKD method achieves comparable results with other SOTA knowledge distillation algorithms for style transfer. Besides, the proposed KBStyle achieves high-quality stylized images. And the inference time of the KBStyle on an Nvidia TITAN RTX GPU is only 20 ms when the resolutions of the content image and style image are both 2k-resolution (2048×1080). Moreover, the 200 KB model size of KBStyle is much smaller than the SOTA models and facilitates style transfer on mobile devices.

Moment-Consistent Contrastive CycleGAN for Cross-Domain Pancreatic Image Segmentation.

CT and MR are currently the most common imaging techniques for pancreatic cancer diagnosis. Accurate segmentation of the pancreas in CT and MR images can provide significant help in the diagnosis and treatment of pancreatic cancer. Traditional supervised segmentation methods require a large number of labeled CT and MR training data, which is usually time-consuming and laborious. Meanwhile, due to domain shift, traditional segmentation networks are difficult to be deployed on different imaging modality datasets. Cross-domain segmentation can utilize labeled source domain data to assist unlabeled target domains in solving the above problems. In this paper, a cross-domain pancreas segmentation algorithm is proposed based on Moment-Consistent Contrastive Cycle Generative Adversarial Networks (MC-CCycleGAN). MC-CCycleGAN is a style transfer network, in which the encoder of its generator is used to extract features from real images and style transfer images, constrain feature extraction through a contrastive loss, and fully extract structural features of input images during style transfer while eliminate redundant style features. The multi-order central moments of the pancreas are proposed to describe its anatomy in high dimensions and a contrastive loss is also proposed to constrain the moment consistency, so as to maintain consistency of the pancreatic structure and shape before and after style transfer. Multi-teacher knowledge distillation framework is proposed to transfer the knowledge from multiple teachers to a single student, so as to improve the robustness and performance of the student network. The experimental results have demonstrated the superiority of our framework over state-of-the-art domain adaptation methods.

Kiosk 10R-TB-08 - Automated Stress Perfusion Prognostic Assessment Using Style Transfer, Segmentation and Hybrid Neural Network for Mortality Prediction

Kiosk 10R-TB-08 - Automated Stress Perfusion Prognostic Assessment Using Style Transfer, Segmentation and Hybrid Neural Network for Mortality Prediction

High Resolution Controllable Portrait Video Style Transfer Network Based on Improved StyleGAN

High Resolution Controllable Portrait Video Style Transfer Network Based on Improved StyleGAN

UPST-NeRF: Universal Photorealistic Style Transfer of Neural Radiance Fields for 3D Scene.

Photorealistic stylization of 3D scenes aims to generate photorealistic images from arbitrary novel views according to a given style image, while ensuring consistency when rendering video from different viewpoints. Some existing stylization methods using neural radiance fields can effectively predict stylized scenes by combining the features of the style image with multi-view images to train 3D scenes. However, these methods generate novel view images that contain undesirable artifacts. In addition, they cannot achieve universal photorealistic stylization for a 3D scene. Therefore, a stylization image needs to retrain a 3D scene representation network based on a neural radiation field. We propose a novel photorealistic 3D scene stylization transfer framework to address these issues. It can realize photorealistic 3D scene style transfer with a 2D style image for novel view video rendering. We first pre-trained a 2D photorealistic style transfer network, which can satisfy the photorealistic style transfer between any content image and style image. Then, we use voxel features to optimize a 3D scene and obtain the geometric representation of the scene. Finally, we jointly optimize a hypernetwork to realize the photorealistic style transfer of arbitrary style images. In the transfer stage, we use a pre-trained 2D photorealistic network to constrain the photorealistic style of different views and different style images in the 3D scene. The experimental results show that our method not only realizes the 3D photorealistic style transfer of arbitrary style images, but also outperforms the existing methods in terms of visual quality and consistency. Project page:https://semchan.github.io/UPST_NeRF/.

Physical adversarial attack in artificial intelligence of things

AbstractWith the continuous development of wireless communication and artificial intelligence technology, Internet of Things (IoT) technology has made great progress. Deep learning methods are currently used in IoT technology, but deep neural networks (DNNs) are notoriously susceptible to adversarial examples, and subtle pixel changes to images can result in incorrect recognition results from DNNs. In the real‐world application, the patches generated by the recent physical attack methods are larger or less realistic and easily detectable. To address this problem, a Generative Adversarial Network based on Visual attention model and Style transfer network (GAN‐VS) is proposed, which reduces the patch area and makes the patch more natural and less noticeable. A visual attention model combined with generative adversarial network is introduced to detect the critical regions of image recognition, and only generate patches within the critical regions to reduce patch area and improve attack efficiency. For any type of seed patch, an adversarial patch can be generated with a high degree of stylistic and content similarity to the attacked image by generative adversarial network and style transfer network. Experimental evaluation shows that the proposed GAN‐VS has good camouflage and outperforms state‐of‐the‐art adversarial patch attack methods.

DualGGAN: A New Facial Style Transfer Network

In response to the background penetration problem of unsupervised style transfer algorithms in most cases, a Transformer style transfer network DualGGAN based on dual generators and fusion of relative position encoding is proposed. The network is trained using the least squares generative adversarial network, and the neural network is used as the image feature extractor to generate feature maps to obtain facial image features with attention weights from the feature maps, utilizing relative position encoding and mask loss to jointly constrain feature region style transfer. The experimental results show that the DualGGAN network effectively reduces artifact generation when implementing facial style transfer, maintains good background consistency, and has good generalization ability. Experiments have shown that the FID and KID indicators on the cat2dog and older2adult datasets are significantly improved compared to other algorithms.

A New Lightweight Script Independent Scene Text Style Transfer Network

Scene text style transfer without a language barrier is an open challenge for the video and scene text recognition community because this plays a vital role in poster, web design, augmenting character images, and editing characters to improve scene text recognition performance and usability. This work presents a new model, called Script Independent Scene Text Style Transfer Network (SISTSTNet), for extracting scene characters and transferring text style simultaneously. The SISTSTNet performs mapping in language-independent feature space for transferring style. It is designed based on a Style Parameter Network and Target Encoder Network through lightweight MobileNetv3 convolutional and residual blocks to capture the style and shape to generate target characters. Similarly, a generative model is explored through the Visual Geometry Group (VGG) network for character replacement. The SISTSTNet is flexible and works on different languages and arbitrary examples in a neat and unified fashion. The experimental results on images in various languages, namely, English, Chinese, Hindi, Russian, Japanese, Arabic, Greek, and Bengali and cross-language validation demonstrate the effectiveness of the proposed method. The performance of the method is superior compared to the state-of-the-art methods in terms of quality measures, language independence, shape-preserving, and efficiency. The code and dataset will be released to the public to support reproducibility.

The Generation of Articulatory Animations Based on Keypoint Detection and Motion Transfer Combined with Image Style Transfer

Knowing the correct positioning of the tongue and mouth for pronunciation is crucial for learning English pronunciation correctly. Articulatory animation is an effective way to address the above task and helpful to English learners. However, articulatory animations are all traditionally hand-drawn. Different situations require varying animation styles, so a comprehensive redraw of all the articulatory animations is necessary. To address this issue, we developed a method for the automatic generation of articulatory animations using a deep learning system. Our method leverages an automatic keypoint-based detection network, a motion transfer network, and a style transfer network to generate a series of articulatory animations that adhere to the desired style. By inputting a target-style articulation image, our system is capable of producing animations with the desired characteristics. We created a dataset of articulation images and animations from public sources, including the International Phonetic Association (IPA), to establish our articulation image animation dataset. We performed preprocessing on the articulation images by segmenting them into distinct areas each corresponding to a specific articulatory part, such as the tongue, upper jaw, lower jaw, soft palate, and vocal cords. We trained a deep neural network model capable of automatically detecting the keypoints in typical articulation images. Also, we trained a generative adversarial network (GAN) model that can generate end-to-end animation of different styles automatically from the characteristics of keypoints and the learned image style. To train a relatively robust model, we used four different style videos: one magnetic resonance imaging (MRI) articulatory video and three hand-drawn videos. For further applications, we combined the consonant and vowel animations together to generate a syllable animation and the animation of a word consisting of many syllables. Experiments show that this system can auto-generate articulatory animations according to input phonetic symbols and should be helpful to people for English articulation correction.

Neural Rendering-Based 3D Scene Style Transfer Method via Semantic Understanding Using a Single Style Image

In the rapidly emerging era of untact (“contact-free”) technologies, the requirement for three-dimensional (3D) virtual environments utilized in virtual reality (VR)/augmented reality (AR) and the metaverse has seen significant growth, owing to their extensive application across various domains. Current research focuses on the automatic transfer of the style of rendering images within a 3D virtual environment using artificial intelligence, which aims to minimize human intervention. However, the prevalent studies on rendering-based 3D environment-style transfers have certain inherent limitations. First, the training of a style transfer network dedicated to 3D virtual environments demands considerable style image data. These data must align with viewpoints that closely resemble those of the virtual environment. Second, there was noticeable inconsistency within the 3D structures. Predominant studies often neglect 3D scene geometry information instead of relying solely on 2D input image features. Finally, style adaptation fails to accommodate the unique characteristics inherent in each object. To address these issues, we propose a novel approach: a neural rendering-based 3D scene-style conversion technique. This methodology employs semantic nearest-neighbor feature matching, thereby facilitating the transfer of style within a 3D scene while considering the distinctive characteristics of each object, even when employing a single style image. The neural radiance field enables the network to comprehend the geometric information of a 3D scene in relation to its viewpoint. Subsequently, it transfers style features by employing the unique features of a single style image via semantic nearest-neighbor feature matching. In an empirical context, our proposed semantic 3D scene style transfer method was applied to 3D scene style transfers for both interior and exterior environments. This application utilizes the replica, 3DFront, and Tanks and Temples datasets for testing. The results illustrate that the proposed methodology surpasses existing style transfer techniques in terms of maintaining 3D viewpoint consistency, style uniformity, and semantic coherence.

A novel intrarenal multimodal 2D/3D registration algorithm and preliminary phantom study.

Retrograde intrarenal surgery (RIRS) is a widely utilized diagnostic and therapeutic tool for multiple upper urinary tract pathologies. The image-guided navigation system can assist the surgeon to perform precise surgery by providing the relative position between the lesion and the instrument after the intraoperative image is registered with the preoperative model. However, due to the structural complexity and diversity of multi-branched organs such as kidneys, bronchi, etc., the consistency of the intensity distribution of virtual and real images will be challenged, which makes the classical pure intensity registration method prone to bias and random results in a wide search domain. In this paper, we propose a structural feature similarity-based method combined with a semantic style transfer network, which significantly improves the registration accuracy when the initial state deviation is obvious. Furthermore, multi-view constraints are introduced to compensate for the collapse of spatial depth information and improve the robustness of the algorithm. Experimental studies were conducted on two models generated from patient data to evaluate the performance of the method and competing algorithms. The proposed method obtains mean target error (mTRE) of 0.971±0.585mm and 1.266±0.416mm respectively, with better accuracy and robustness overall. Experimental results demonstrate that the proposed method has the potential to be applied to RIRS and extended to other organs with similar structures.