Image Style Research Articles

Neural Style Transfer, Creating Art with Deep Learning

Abstract—Neural style transfer is a deep learning technique that is the topic of study in this research paper. In NST two images are combined, a content and a style image. Where, while retaining the content image the artistic style of style image is imposed on the content image. In this research, a series of ways are explored to improve the efficiency and visual quality of the generated image. This research is about improvising and molding the existing losses, to improve the existing methodologies. Key contributions are dynamic weighting of content and style losses, multi scale loss computation for preserving the details in a better way. This kind of loss improvisation and changing is being used, so that not only the high-level structures and fine details are maintained in the image generated. These dynamic and multiple kind of losses will be implemented to retain the essence of the content in the content image. While generating a new image the content or style is overcompensated and most of the time one of these is having a stronger effect on the final image in most of the techniques, our motive is to eradicate the same and generate an excellent image. Along with this, various optimization techniques are studied in the underlying paper to compensate this computation cost of the newly introduced losses. The ways provide a framework for a neural style transfer with a higher quality and efficient combining of images. Index Terms—NST (Neural style transfer), dynamic weighting, multi-scale losses.

Research on the Principles and Applications of Generative Adversarial Networks in Image Style Transfer

Abstract. With the rapid development of machine learning, the field of image style transfer has received widespread attention, and scientists have gradually realized that this is an important field in computer vision. Image style transfer refers to the conversion of an image from one visual style to another and preserving the content. The technology of changing image style has a wide range of applications in various fields from art to virtual reality. Although current learning models have achieved some results in the study of image style transfer, they still face challenges in improving resolution and complex style transfer. This paper mainly introduces the background and characteristics of existing learning models that can achieve image style transfer. It introduces the principles, characteristics and limitations of GANs. In the basis of introduction, this paper also analyses the differences and different characteristics of GANs and summarizes the advantages and disadvantages of GANs. At the same time, this paper predicts future research directions, continuing to optimize these models, reducing training costs, and exploring hybrid methods to further advance image style transfer technology. These developments not only strengthen the theoretical research of image style transfer and GANs, but also have broad application potential in fields such as art, design and medical imaging.

SAFENet: Semantic-Aware Feature Enhancement Network for unsupervised cross-domain road scene segmentation

Unsupervised cross-domain road scene segmentation has attracted substantial interest because of its capability to perform segmentation on new and unlabeled domains, thereby reducing the dependence on expensive manual annotations. This is achieved by leveraging networks trained on labeled source domains to classify images on unlabeled target domains. Conventional techniques usually use adversarial networks to align inputs from the source and the target in either of their domains. However, these approaches often fall short in effectively integrating information from both domains due to Alignment in each space usually leads to bias problems during feature learning. To overcome these limitations and enhance cross-domain interaction while mitigating overfitting to the source domain, we introduce a novel framework called Semantic-Aware Feature Enhancement Network (SAFENet) for Unsupervised Cross-domain Road Scene Segmentation. SAFENet incorporates the Semantic-Aware Enhancement (SAE) module to amplify the importance of class information in segmentation tasks and uses the semantic space as a new domain to guide the alignment of the source and target domains. Additionally, we integrate Adaptive Instance Normalization with Momentum (AdaIN-M) techniques, which convert the source domain image style to the target domain image style, thereby reducing the adverse effects of source domain overfitting on target domain segmentation performance. Moreover, SAFENet employs a Knowledge Transfer (KT) module to optimize network architecture, enhancing computational efficiency during testing while maintaining the robust inference capabilities developed during training. To further improve the segmentation performance, we further employ Curriculum Learning, a self-training mechanism that uses pseudo-labels derived from the target domain to iteratively refine the network. Comprehensive experiments on three well-known datasets, “Synthia→Cityscapes” and “GTA5→Cityscapes”, demonstrate the superior performance of our method. In-depth examinations and ablation studies verify the efficacy of each module within the proposed method.

Style harmonization of panoramic radiography using deep learning.

This study aimed to harmonize panoramic radiograph images from different equipment in a single institution to display similar styles. A total of 15,624 panoramic images were acquired using two different equipment: 8079 images from Rayscan Alpha Plus (R-unit) and 7545 images from Pax-i plus (P-unit). Among these, 222 image pairs (444 images) from the same patients comprised the test dataset to harmonize the P-unit images with the R-unit image style using CycleGAN. Objective evaluations included Frechet Inception Distance (FID) and Learned Perceptual Image Patch Similarity (LPIPS) assessments. Additionally, expert evaluation was conducted by two oral and maxillofacial radiologists on transformed P-unit and R-unit images. The statistical analysis of LPIPS employed a Student's t-test. The FID and mean LPIPS values of the transformed P-unit images (7.362, 0.488) were lower than those of the original P-unit images (8.380, 0.519), with a significant difference in LPIPS (p < 0.05). The experts evaluated 43.3-46.7% of the transformed P-unit images as R-unit images, 20.0-28.3% as P-units, and 28.3-33.3% as undetermined images. CycleGAN has the potential to harmonize panoramic radiograph image styles. Enhancement of the model is anticipated for the application of images produced by additional units.

Research on Image Style Transfer Based on a Generative Model

The generation of artificial intelligence technology is rapidly developing. As a representative, Sora created in text, image rendering, and video production shows excellent ability. This article discusses the image-style transmission algorithm based on generating a confrontation network (GAN) and compares it with the Sora model. Studies have shown that GAN performs well in image style transfer, especially in maintaining the consistency of the image content and generating high-quality images. In the future, the expected artificial intelligence technology will further improve the quality and diversity of generating content by combining the diffusion model and GAN and expanding its application potential in the fields of art creation and virtual reality.

Stealthy Vehicle Adversarial Camouflage Texture Generation Based on Neural Style Transfer

Adversarial attacks that mislead deep neural networks (DNNs) into making incorrect predictions can also be implemented in the physical world. However, most of the existing adversarial camouflage textures that attack object detection models only consider the effectiveness of the attack, ignoring the stealthiness of adversarial attacks, resulting in the generated adversarial camouflage textures appearing abrupt to human observers. To address this issue, we propose a style transfer module added to an adversarial texture generation framework. By calculating the style loss between the texture and the specified style image, the adversarial texture generated by the model is guided to have good stealthiness and is not easily detected by DNNs and human observers in specific scenes. Experiments have shown that in both the digital and physical worlds, the vehicle full coverage adversarial camouflage texture we create has good stealthiness and can effectively fool advanced DNN object detectors while evading human observers in specific scenes.

Automated tree crown labeling with 3D radiative transfer modelling achieves human comparable performances for tree segmentation in semi-arid landscapes

Mapping tree crowns in arid or semi-arid areas, which cover around one-third of the Earth’s land surface, is a key methodology towards sustainable management of trees. Recent advances in deep learning have shown promising results for tree crown segmentation. However, a large amount of manually labeled data is still required. We here propose a novel method to delineate tree crowns from high resolution satellite imagery using deep learning trained with automatically generated labels from 3D radiative transfer modeling, intending to reduce human annotation significantly. The methodological steps consist of 1) simulating images with a 3D radiative transfer model, 2) image style transfer learning based on generative adversarial network (GAN) and 3) tree crown segmentation using U-net segmentation model. The delineation performances of the proposed method have been evaluated on a manually annotated dataset consisting of more than 40,000 tree crowns. Our approach, which relies solely on synthetic images, demonstrates high segmentation accuracy, with an F1 score exceeding 0.77 and an Intersection over Union (IoU) above 0.64. Particularly, it achieves impressive accuracy in extracting crown areas (r2 greater than 0.87) and crown densities (r2 greater than 0.72), comparable to that of a trained dataset with human annotations only. In this study, we demonstrated that the integration of a 3D radiative transfer model and GANs for automatically generating training labels can achieve performances comparable to human labeling, and can significantly reduce the time needed for manual labeling in remote sensing segmentation applications.

A Study on Enhancing the Visual Fidelity of Aviation Simulators Using WGAN-GP for Remote Sensing Image Color Correction

When implementing outside-the-window (OTW) visuals in aviation tactical simulators, maintaining terrain image color consistency is critical for enhancing pilot immersion and focus. However, due to various environmental factors, inconsistent image colors in terrain can cause visual confusion and diminish realism. To address these issues, a color correction technique based on a Wasserstein Generative Adversarial Network with Gradient Penalty (WGAN-GP) is proposed. The proposed WGAN-GP model utilizes multi-scale feature extraction and Wasserstein distance to effectively measure and adjust the color distribution difference between the input image and the reference image. This approach can preserve the texture and structural characteristics of the image while maintaining color consistency. In particular, by converting Bands 2, 3, and 4 of the BigEarthNet-S2 dataset into RGB images as the reference image and preprocessing the reference image to serve as the input image, it is demonstrated that the proposed WGAN-GP model can handle large-scale remote sensing images containing various lighting conditions and color differences. The experimental results showed that the proposed WGAN-GP model outperformed traditional methods, such as histogram matching and color transfer, and was effective in reflecting the style of the reference image to the target image while maintaining the structural elements of the target image during the training process. Quantitative analysis demonstrated that the mid-stage model achieved a PSNR of 28.93 dB and an SSIM of 0.7116, which significantly outperforms traditional methods. Furthermore, the LPIPS score was reduced to 0.3978, indicating improved perceptual similarity. This approach can contribute to improving the visual elements of the simulator to enhance pilot immersion and has the potential to significantly reduce time and costs compared to the manual methods currently used by the Republic of Korea Air Force.

Generating Artistic Images Through Neural Style Transfer Using Machine Learning and Image Manipulation Techniques

Neural Style Transfer (NST) is a computational technique that merges the content of one image with the stylistic characteristics of another, creating visually appealing compositions. Leveraging deep learning and image processing, this method separates andcombines the content and style of input images using convolutional neural networks (CNNs). The content representation of an image is captured by the high-level features extracted from a pre-trained CNN, while its style is characterized by the statistical correlations of features at multiple network layers. It explores the fusion of machine learning and image processing techniques in NST. Initially introduced by Gatys et al., NST has gained popularity due to its ability to generate artistic renditions and novel visualizations. The process involves extracting content and style features from separate images and optimizing a generated image to minimize differences in content and style representations. In this work, explored into the underlying concepts and technical aspectsof NST. This proceedings discussed the utilization of pre-trained CNN models like VGG19 or ResNet for feature extraction and how optimization algorithms, such as gradient descent, iteratively refine the generated image to achieve a fusion of content and style. Additionally, This development has been examined the impact of hyper parameters and layer choices on the quality of stylized outputs.Mainly this technology focuses on reduction of total variation loss, which we incur during the style transfer.These proceedings has carried on work with VGG19, XCEPTION, EfficientnetB7. These findings show how Convolutional Neural Networks can learn deep image representations and show how they may be used for sophisticated image synthesis andmodification.

Semi-Supervised Interior Decoration Style Classification with Contrastive Mutual Learning

Precisely identifying interior decoration styles holds substantial significance in directing interior decoration practices. Nevertheless, constructing accurate models for the automatic classification of interior decoration styles remains challenging due to the scarcity of expert annotations. To address this problem, we propose a novel pseudo-label-guided contrastive mutual learning framework (PCML) for semi-supervised interior decoration style classification by harnessing large amounts of unlabeled data. Specifically, PCML introduces two distinct subnetworks and selectively utilizes the diversified pseudo-labels generated by each for mutual supervision, thereby mitigating the issue of confirmation bias. For labeled images, the inconsistent pseudo-labels generated by the two subnetworks are employed to identify images that are prone to misclassification. We then devise an inconsistency-aware relearning (ICR) regularization model to perform a review training process. For unlabeled images, we introduce a class-aware contrastive learning (CCL) regularization to learn their discriminative feature representations using the corresponding pseudo-labels. Since the use of distinct subnetworks reduces the risk of both models producing identical erroneous pseudo-labels, CCL can reduce the possibility of noise data sampling to enhance the effectiveness of contrastive learning. The performance of PCML is evaluated on five interior decoration style image datasets. For the average AUC, accuracy, sensitivity, specificity, precision, and F1 scores, PCML obtains improvements of 1.67%, 1.72%, 3.65%, 1.0%, 4.61%, and 4.66% in comparison with the state-of-the-art method, demonstrating the superiority of our method.

Local part attention for image stylization with text prompt

Local part attention for image stylization with text prompt

ST2SI: Image Style Transfer via Vision Transformer using Spatial Interaction

While retaining the original content structure, image style transfer uses style image to render it to obtain stylized images with artistic features. Because the content image contains different detail units and the style image has various style patterns, it is easy to cause the distortion of the stylized image. We proposes a new Style Transfer based on Vision Transformer using Spatial Interaction (ST2SI), which takes advantage of Spatial Interactive Convolution (SIC) and Spatial Unit Attention (SUA) to further enhance the content and style representation, so that the encoder can not only better learn the features of the content domain and the style domain, but also maintain the structural integrity of the image content and the effective integration of style features. Concretely, the high-order spatial interaction ability of Spatial Interactive Convolution can capture complex style patterns, and Spatial Unit Attention can balance the content information of different detail units through the change of attention weight, thus solving the problem of image distortion. Comprehensive qualitative and quantitative experiments prove the efficacy of our approach.

Enhancing Inter-AUV Perception: Adaptive 6-DOF Pose Estimation with Synthetic Images for AUV Swarm Sensing

The capabilities of AUV mutual perception and localization are crucial for the development of AUV swarm systems. We propose the AUV6D model, a synthetic image-based approach to enhance inter-AUV perception through 6D pose estimation. Due to the challenge of acquiring accurate 6D pose data, a dataset of simulated underwater images with precise pose labels was generated using Unity3D. Mask-CycleGAN technology was introduced to transform these simulated images into realistic synthetic images, addressing the scarcity of available underwater data. Furthermore, the Color Intermediate Domain Mapping strategy is proposed to ensure alignment across different image styles at pixel and feature levels, enhancing the adaptability of the pose estimation model. Additionally, the Salient Keypoint Vector Voting Mechanism was developed to improve the accuracy and robustness of underwater pose estimation, enabling precise localization even in the presence of occlusions. The experimental results demonstrated that our AUV6D model achieved millimeter-level localization precision and pose estimation errors within five degrees, showing exceptional performance in complex underwater environments. Navigation experiments with two AUVs further verified the model’s reliability for mutual 6D pose estimation. This research provides substantial technical support for more complex and precise collaborative operations for AUV swarms in the future.

Deep Learning-based Unsupervised Domain Adaptation via a Unified Model for Prostate Lesion Detection Using Multisite Biparametric MRI Datasets.

Purpose To determine whether the unsupervised domain adaptation (UDA) method with generated images improves the performance of a supervised learning (SL) model for prostate cancer (PCa) detection using multisite biparametric (bp) MRI datasets. Materials and Methods This retrospective study included data from 5150 patients (14 191 samples) collected across nine different imaging centers. A novel UDA method using a unified generative model was developed for PCa detection using multisite bpMRI datasets. This method translates diffusion-weighted imaging (DWI) acquisitions, including apparent diffusion coefficient (ADC) and individual diffusion-weighted (DW) images acquired using various b values, to align with the style of images acquired using b values recommended by Prostate Imaging Reporting and Data System (PI-RADS) guidelines. The generated ADC and DW images replace the original images for PCa detection. An independent set of 1692 test cases (2393 samples) was used for evaluation. The area under the receiver operating characteristic curve (AUC) was used as the primary metric, and statistical analysis was performed via bootstrapping. Results For all test cases, the AUC values for baseline SL and UDA methods were 0.73 and 0.79 (P < .001), respectively, for PCa lesions with PI-RADS score of 3 or greater and 0.77 and 0.80 (P < .001) for lesions with PI-RADS scores of 4 or greater. In the 361 test cases under the most unfavorable image acquisition setting, the AUC values for baseline SL and UDA were 0.49 and 0.76 (P < .001) for lesions with PI-RADS scores of 3 or greater and 0.50 and 0.77 (P < .001) for lesions with PI-RADS scores of 4 or greater. Conclusion UDA with generated images improved the performance of SL methods in PCa lesion detection across multisite datasets with various b values, especially for images acquired with significant deviations from the PI-RADS-recommended DWI protocol (eg, with an extremely high b value). Keywords: Prostate Cancer Detection, Multisite, Unsupervised Domain Adaptation, Diffusion-weighted Imaging, b Value Supplemental material is available for this article. © RSNA, 2024.

Exploration of Art Digital Design Technology Guided by Visual Communication Elements

From the perspective of visual communication elements, this paper proposes a digital design composite model based on style migration, trying to explore and innovate art digital design technology. In the image recognition and style migration model, the attention mechanism is added on the basis of ResNet50 network model to enhance the image recognition effect, and on the basis of VGG19 deep convolutional network model and optimization algorithm to realize the image style migration technology and digital redesign. On the basis of the proposed style migration model, a graphic rendering model is further proposed by combining the GrabCut algorithm and the VGG16 neural network, so as to realize the local style migration of patterns. In the application analysis of the artistic digitization of Zhuang brocade lamps, for example, the total percentage of “very approved” and “more approved” in the assessment of the style migration effect is more than 30%, and the style migration effect is better. The average scores of the art design effect evaluation of layers, elements and color richness are 4.02, 4.15 and 4.31, which are all higher than 4 points, and the performance is good. It is also recognized by experts and ordinary respondents in the satisfaction evaluation of art design.

Cyclic Consistent Image Style Transformation: From Model to System

Cyclic Consistent Image Style Transformation: From Model to System

HAIGEN: Towards Human-AI Collaboration for Facilitating Creativity and Style Generation in Fashion Design

The process of fashion design usually involves sketching, refining, and coloring, with designers drawing inspiration from various images to fuel their creative endeavors. However, conventional image search methods often yield irrelevant results, impeding the design process. Moreover, creating and coloring sketches can be time-consuming and demanding, acting as a bottleneck in the design workflow. In this work, we introduce HAIGEN (Human-AI Collaboration for GENeration), an efficient fashion design system for Human-AI collaboration developed to aid designers. Specifically, HAIGEN consists of four modules. T2IM, located in the cloud, generates reference inspiration images directly from text prompts. With three other modules situated locally, the I2SM batch generates the image material library into a certain designer-style sketch material library. The SRM recommends similar sketches in the generated library to designers for further refinement, and the STM colors the refined sketch according to the styles of inspiration images. Through our system, any designer can perform local personalized fine-tuning and leverage the powerful generation capabilities of large models in the cloud, streamlining the entire design development process. Given that our approach integrates both cloud and local model deployment schemes, it effectively safeguards design privacy by avoiding the need to upload personalized data from local designers. We validated the effectiveness of each module through extensive qualitative and quantitative experiments. User surveys also confirmed that HAIGEN offers significant advantages in design efficiency, positioning it as a new generation of aid-tool for designers.

Audio-guided implicit neural representation for local image stylization

We present a novel framework for audio-guided localized image stylization. Sound often provides information about the specific context of a scene and is closely related to a certain part of the scene or object. However, existing image stylization works have focused on stylizing the entire image using an image or text input. Stylizing a particular part of the image based on audio input is natural but challenging. This work proposes a framework in which a user provides an audio input to localize the target in the input image and another to locally stylize the target object or scene. We first produce a fine localization map using an audio-visual localization network leveraging CLIP embedding space. We then utilize an implicit neural representation (INR) along with the predicted localization map to stylize the target based on sound information. The INR manipulates local pixel values to be semantically consistent with the provided audio input. Our experiments show that the proposed framework outperforms other audio-guided stylization methods. Moreover, we observe that our method constructs concise localization maps and naturally manipulates the target object or scene in accordance with the given audio input.

Image style migration method based on transformer generative adversarial network combined with contrast learning strategy

Abstract Aiming at the problems of large loss of image patch embedding information and lack of stylized features in the generated image during transformer style migration, a style migration model combining contrast learning with transformer backbone generative adversarial network is established. Firstly, overlapping patch embedding is used to serialize the input images to get richer features; then the style and content contrast loss of contrast learning is used to mine the non-significant information between stylized images to reduce the style and content loss; finally, a multi-scale discriminator discriminates the true and false of stylized images to optimize the generator to get more stylistic features. The results on MS-COCO and WikiArt datasets show that the model improves the effectiveness and transformation efficacy of stylized images in style migration.

A deep learning framework for predicting endometrial cancer from cytopathologic images with different staining styles.

Endometrial cancer screening is crucial for clinical treatment. Currently, cytopathologists analyze cytopathology images is considered a popular screening method, but manual diagnosis is time-consuming and laborious. Deep learning can provide objective guidance efficiency. But endometrial cytopathology images often come from different medical centers with different staining styles. It decreases the generalization ability of deep learning models in cytopathology images analysis, leading to poor performance. This study presents a robust automated screening framework for endometrial cancer that can be applied to cytopathology images with different staining styles, and provide an objective diagnostic reference for cytopathologists, thus contributing to clinical treatment. We collected and built the XJTU-EC dataset, the first cytopathology dataset that includes segmentation and classification labels. And we propose an efficient two-stage framework for adapting different staining style images, and screening endometrial cancer at the cellular level. Specifically, in the first stage, a novel CM-UNet is utilized to segment cell clumps, with a channel attention (CA) module and a multi-level semantic supervision (MSS) module. It can ignore staining variance and focus on extracting semantic information for segmentation. In the second stage, we propose a robust and effective classification algorithm based on contrastive learning, ECRNet. By momentum-based updating and adding labeled memory banks, it can reduce most of the false negative results. On the XJTU-EC dataset, CM-UNet achieves an excellent segmentation performance, and ECRNet obtains an accuracy of 98.50%, a precision of 99.32% and a sensitivity of 97.67% on the test set, which outperforms other competitive classical models. Our method robustly predicts endometrial cancer on cytopathologic images with different staining styles, which will further advance research in endometrial cancer screening and provide early diagnosis for patients. The code will be available on GitHub.