Style Transfer Research Articles

Advancements in Data Augmentation and Transfer Learning: A Comprehensive Survey to Address Data Scarcity Challenges

Abstract: Deep Learning (DL) models have demonstrated remarkable proficiency in image classification and recognition tasks, surpassing human capabilities. The observed enhancement in performance can be attributed to the utilization of extensive datasets. Nevertheless, DL models have huge data requirements. Widening the learning capability of such models from limited samples even today remains a challenge, given the intrinsic constraints of small datasets. The trifecta of challenges, encompassing limited labeled datasets, privacy, poor generalization performance, and the costliness of annotations, further compounds the difficulty in achieving robust model performance. Overcoming the challenge of expanding the learning capabilities of Deep Learning models with limited sample sizes remains a pressing concern even today. To address this critical issue, our study conducts a meticulous examination of established methodologies, such as Data Augmentation and Transfer Learning, which offer promising solutions to data scarcity dilemmas. Data Augmentation, a powerful technique, amplifies the size of small datasets through a diverse array of strategies. These encompass geometric transformations, kernel filter manipulations, neural style transfer amalgamation, random erasing, Generative Adversarial Networks, augmentations in feature space, and adversarial and meta- learning training paradigms. : Furthermore, Transfer Learning emerges as a crucial tool, leveraging pre-trained models to facilitate knowledge transfer between models or enabling the retraining of models on analogous datasets. Through our comprehensive investigation, we provide profound insights into how the synergistic application of these two techniques can significantly enhance the performance of classification tasks, effectively magnifying scarce datasets. This augmentation in data availability not only addresses the immediate challenges posed by limited datasets but also unlocks the full potential of working with Big Data in a new era of possibilities in DL applications.

Stealthy Vehicle Adversarial Camouflage Texture Generation Based on Neural Style Transfer

Adversarial attacks that mislead deep neural networks (DNNs) into making incorrect predictions can also be implemented in the physical world. However, most of the existing adversarial camouflage textures that attack object detection models only consider the effectiveness of the attack, ignoring the stealthiness of adversarial attacks, resulting in the generated adversarial camouflage textures appearing abrupt to human observers. To address this issue, we propose a style transfer module added to an adversarial texture generation framework. By calculating the style loss between the texture and the specified style image, the adversarial texture generated by the model is guided to have good stealthiness and is not easily detected by DNNs and human observers in specific scenes. Experiments have shown that in both the digital and physical worlds, the vehicle full coverage adversarial camouflage texture we create has good stealthiness and can effectively fool advanced DNN object detectors while evading human observers in specific scenes.

GVVST: Image-Driven Style Extraction From Graph Visualizations for Visual Style Transfer.

Incorporating automatic style extraction and transfer from existing well-designed graph visualizations can significantly alleviate the designer's workload. There are many types of graph visualizations. In this paper, our work focuses on node-link diagrams. We present a novel approach to streamline the design process of graph visualizations by automatically extracting visual styles from well-designed examples and applying them to other graphs. Our formative study identifies the key styles that designers consider when crafting visualizations, categorizing them into global and local styles. Leveraging deep learning techniques such as saliency detection models and multi-label classification models, we develop end-to-end pipelines for extracting both global and local styles. Global styles focus on aspects such as color scheme and layout, while local styles are concerned with the finer details of node and edge representations. Through a user study and evaluation experiment, we demonstrate the efficacy and time-saving benefits of our method, highlighting its potential to enhance the graph visualization design process.

Automated tree crown labeling with 3D radiative transfer modelling achieves human comparable performances for tree segmentation in semi-arid landscapes

Mapping tree crowns in arid or semi-arid areas, which cover around one-third of the Earth’s land surface, is a key methodology towards sustainable management of trees. Recent advances in deep learning have shown promising results for tree crown segmentation. However, a large amount of manually labeled data is still required. We here propose a novel method to delineate tree crowns from high resolution satellite imagery using deep learning trained with automatically generated labels from 3D radiative transfer modeling, intending to reduce human annotation significantly. The methodological steps consist of 1) simulating images with a 3D radiative transfer model, 2) image style transfer learning based on generative adversarial network (GAN) and 3) tree crown segmentation using U-net segmentation model. The delineation performances of the proposed method have been evaluated on a manually annotated dataset consisting of more than 40,000 tree crowns. Our approach, which relies solely on synthetic images, demonstrates high segmentation accuracy, with an F1 score exceeding 0.77 and an Intersection over Union (IoU) above 0.64. Particularly, it achieves impressive accuracy in extracting crown areas (r2 greater than 0.87) and crown densities (r2 greater than 0.72), comparable to that of a trained dataset with human annotations only. In this study, we demonstrated that the integration of a 3D radiative transfer model and GANs for automatically generating training labels can achieve performances comparable to human labeling, and can significantly reduce the time needed for manual labeling in remote sensing segmentation applications.

Appearance-Based Gaze Estimation as a Benchmark for Eye Image Data Generation Methods

Data augmentation is commonly utilized to increase the size and diversity of training sets for deep learning tasks. In this study, we propose a novel application of an existing image generation approach in the domain of realistic eye images that leverages data collected from 40 subjects. This hybrid method combines the benefits of precise control over the image content provided by 3D rendering, while introducing the previously lacking photorealism and diversity into synthetic images through neural style transfer. We prove its general efficacy as a data augmentation tool for appearance-based gaze estimation when generated data are mixed with a sparse train set of real images. It improved the results for 39 out of 40 subjects, with an 11.22% mean and a 19.75% maximum decrease in gaze estimation error, achieving similar metrics for train and held-out subjects. We release our data repository of eye images with gaze labels used in this work for public access.

The Development and Impact of AI-Generated Content in Contemporary Painting

The paper examines the evolution and impact of generative artificial intelligence (AI) technology in the field of painting, tracing its ascent from the early 21st century. It highlights pivotal developments in AI-generated content (AIGC), starting with Harold Cohen's "AARON" program in the early 1990s, progressing to Yang Zhiping's introduction of generative adversarial networks (GANs) in 2000, and culminating in Robbie Barrat's innovative applications in 2018. These milestones underscore the swift progression and vast potential of AIGC for both technical innovation and artistic expression. This analysis delves into the current landscape of AIGC in painting, identifying key technological advancements such as deep learning, GANs, style transfer, adaptive algorithms, and augmented reality. These technologies have elevated artistic creation to unprecedented levels. However, AIGC also confronts significant challenges, including issues related to copyright and intellectual property, artistic ethics and originality, technical constraints, market acceptance, and socio-ethical and legal concerns. To address these challenges, the paper proposes several solutions: clarifying legal standards, establishing authorization mechanisms, fostering hybrid creative models, enhancing the diversity of training data, promoting public education and outreach, and developing professional training and ethical guidelines.

A Depth Awareness and Learnable Feature Fusion Network for Enhanced Geometric Perception in Semantic Correspondence.

Deep learning is becoming the most widely used technology for multi-sensor data fusion. Semantic correspondence has recently emerged as a foundational task, enabling a range of downstream applications, such as style or appearance transfer, robot manipulation, and pose estimation, through its ability to provide robust correspondence in RGB images with semantic information. However, current representations generated by self-supervised learning and generative models are often limited in their ability to capture and understand the geometric structure of objects, which is significant for matching the correct details in applications of semantic correspondence. Furthermore, efficiently fusing these two types of features presents an interesting challenge. Achieving harmonious integration of these features is crucial for improving the expressive power of models in various tasks. To tackle these issues, our key idea is to integrate depth information from depth estimation or depth sensors into feature maps and leverage learnable weights for feature fusion. First, depth information is used to model pixel-wise depth distributions, assigning relative depth weights to feature maps for perceiving an object's structural information. Then, based on a contrastive learning optimization objective, a series of weights are optimized to leverage feature maps from self-supervised learning and generative models. Depth features are naturally embedded into feature maps, guiding the network to learn geometric structure information about objects and alleviating depth ambiguity issues. Experiments on the SPair-71K and AP-10K datasets show that the proposed method achieves scores of 81.8 and 83.3 on the percentage of correct keypoints (PCK) at the 0.1 level, respectively. Our approach not only demonstrates significant advantages in experimental results but also introduces the depth awareness module and a learnable feature fusion module, which enhances the understanding of object structures through depth information and fully utilizes features from various pre-trained models, offering new possibilities for the application of deep learning in RGB and depth data fusion technologies. We will also continue to focus on accelerating model inference and optimizing model lightweighting, enabling our model to operate at a faster speed.

Object Detection in Historical Images: Transfer Learning and Pseudo Labelling

The automatic analysis of images in the historical sciences often requires the identification of objects. Object identification is a well researched problem for modern photographs, however, for historical material annotations are often necessary. We present a solution for finding objects without manual work. The method consists of a style transfer of images from the COCO dataset into the domain using CycleGAN and training with items obtained through pseudo labelling on the original and the additional transferred COCO images. Different strategies to assemble the dataset are compared. The best method obtains a F1 score of 0.58 for 15 object types without any labelling.

Advances in text-guided 3D editing: a survey

In 3D Artificial Intelligence Generated Content (AIGC), compared with generating 3D assets from scratch, editing extant 3D assets satisfies user prompts, allowing the creation of diverse and high-quality 3D assets in a time and labor-saving manner. More recently, text-guided 3D editing that modifies 3D assets guided by text prompts is user-friendly and practical, which evokes a surge in research within this field. In this survey, we comprehensively investigate recent literature on text-guided 3D editing in an attempt to answer two questions: What are the methodologies of existing text-guided 3D editing? How has current progress in text-guided 3D editing gone so far? Specifically, we focus on text-guided 3D editing methods published in the past 4 years, delving deeply into their frameworks and principles. We then present a fundamental taxonomy in terms of the editing strategy, optimization scheme, and 3D representation. Based on the taxonomy, we review recent advances in this field, considering factors such as editing scale, type, granularity, and perspective. In addition, we highlight four applications of text-guided 3D editing, including texturing, style transfer, local editing of scenes, and insertion editing, to exploit further the 3D editing capacities with in-depth comparisons and discussions. Depending on the insights achieved by this survey, we discuss open challenges and future research directions. We hope this survey will help readers gain a deeper understanding of this exciting field and foster further advancements in text-guided 3D editing.

MapGPT: an autonomous framework for mapping by integrating large language model and cartographic tools

ABSTRACT The mapping process generally involves intricate operations, such as symbol design, layout design, and text annotation, demanding a high level of professional expertise. The high requirement for map producers hinders the promotion and widespread adoption of mapping. Consequently, researchers are concentrating on techniques to automate and enhance the intelligence of the mapping process. For example, some studies attempt to train deep learning models for mapping, including methods like map style transfer. However, these approaches typically treat the entire map as a global input and generate a new map as output, lacking the flexibility to consider and control detailed elements within a map. Therefore, in this paper, we propose a large language model-based intelligent mapping framework, termed MapGPT, which can be used for mapping by considering the map as an integration of various map elements. Specifically, multiple professional mapping tools are designed in MapGPT, and each tool is designed to control a corresponding map element. With these tools, a large language model is used to first understand the demand of users based on mere natural language descriptions, and subsequently automatically invoke appropriate tools in sequence to generate a map. Furthermore, by utilizing a memory component to store interaction information, users can interact with MapGPT through conversation to adjust map elements such as color and position. In conclusion, MapGPT offers user-friendly mapping experience, showing potential to be a mapping assistant for professional map producers. A comprehensive demonstration of this framework is provided in a visual case study video, accessible at https://github.com/AGI-GIS/MapGPT.

Diffusion‐based Human Motion Style Transfer with Semantic Guidance

Abstract3D Human motion style transfer is a fundamental problem in computer graphic and animation processing. Existing AdaIN‐based methods necessitate datasets with balanced style distribution and content/style labels to train the clustered latent space. However, we may encounter a single unseen style example in practical scenarios, but not in sufficient quantity to constitute a style cluster for AdaIN‐based methods. Therefore, in this paper, we propose a novel two‐stage framework for few‐shot style transfer learning based on the diffusion model. Specifically, in the first stage, we pre‐train a diffusion‐based text‐to‐motion model as a generative prior so that it can cope with various content motion inputs. In the second stage, based on the single style example, we fine‐tune the pre‐trained diffusion model in a few‐shot manner to make it capable of style transfer. The key idea is regarding the reverse process of diffusion as a motion‐style translation process since the motion styles can be viewed as special motion variations. During the fine‐tuning for style transfer, a simple yet effective semantic‐guided style transfer loss coordinated with style example reconstruction loss is introduced to supervise the style transfer in CLIP semantic space. The qualitative and quantitative evaluations demonstrate that our method can achieve state‐of‐the‐art performance and has practical applications. The source code is available at https://github.com/hlcdyy/diffusion-based-motion-style-transfer.

Fashion style transfer: enhancing efficiency and aesthetic with a novel two-stage neural network

Neural style transfer is increasingly recognized as a crucial tool in fashion design. It enables the transfer of stylistic elements from an inspirational source image to a fashion product, thus streamlining the design process and enhancing efficiency. However, existing research primarily focuses on the transfer of painting styles, which only partially meets the specific requirements of fashion design. This study introduces a novel two-stage style transfer model for fashion design applications. This model can efficiently transfer fashion patterns that combine global and local features, preserving the original background while achieving symmetry and gradient effects in fashion patterns. Additionally, a five-factor machine scoring system, closely aligned with human preferences, is proposed to provide a more objective evaluation of the application value of style transfer models in fashion design. Our comparative experiments with various models demonstrate that our proposed model meets the specific requirements of fashion design, particularly regarding style transfer efficiency and aesthetic considerations.

Generating Artistic Images Through Neural Style Transfer Using Machine Learning and Image Manipulation Techniques

Neural Style Transfer (NST) is a computational technique that merges the content of one image with the stylistic characteristics of another, creating visually appealing compositions. Leveraging deep learning and image processing, this method separates andcombines the content and style of input images using convolutional neural networks (CNNs). The content representation of an image is captured by the high-level features extracted from a pre-trained CNN, while its style is characterized by the statistical correlations of features at multiple network layers. It explores the fusion of machine learning and image processing techniques in NST. Initially introduced by Gatys et al., NST has gained popularity due to its ability to generate artistic renditions and novel visualizations. The process involves extracting content and style features from separate images and optimizing a generated image to minimize differences in content and style representations. In this work, explored into the underlying concepts and technical aspectsof NST. This proceedings discussed the utilization of pre-trained CNN models like VGG19 or ResNet for feature extraction and how optimization algorithms, such as gradient descent, iteratively refine the generated image to achieve a fusion of content and style. Additionally, This development has been examined the impact of hyper parameters and layer choices on the quality of stylized outputs.Mainly this technology focuses on reduction of total variation loss, which we incur during the style transfer.These proceedings has carried on work with VGG19, XCEPTION, EfficientnetB7. These findings show how Convolutional Neural Networks can learn deep image representations and show how they may be used for sophisticated image synthesis andmodification.

Hiding in Plain Sight: Adversarial Attack via Style Transfer on Image Borders

Hiding in Plain Sight: Adversarial Attack via Style Transfer on Image Borders

Deep Learning Based Method for Dynamic Tracking of Bubbles in Microfluidic Gas-Driven Water Experiments

Using microfluidic chips for gas displacement experiments is an important approach for studying CO2 displacement mechanisms and gas-liquid exchange processes. Analyzing the micro-scale distribution of gas and water during gas displacement enables the optimization of injection strategies, offering valuable guidance for enhancing oil and gas recovery rates. Traditional image processing techniques often face challenges in effectively handle the complex gas-liquid flow dynamics and irregular bubble shapes encountered in gas displacement experiments. To address this challenge, this paper proposes a gas bubble detection and tracking method for gas-driven water microfluidic experiments based on domain adaptation in deep learning and improved YOLOv8. Using the CycleGAN style transfer network to generate numerous simulated bubble training samples significantly reduces the manual labeling workload for bubbles. Additionally, the SE attention mechanism is integrated into the YOLOv8 backbone network to enhance the feature extraction capability of bubbles, and the Fast-PConv Head structure is incorporated into the detection head to enhance detection efficiency. Experimental results demonstrate that using just 20 annotated experimental images, the bubble detection accuracy of the proposed method can reach 94%. Lastly, DeepSORT is employed to track bubbles and visualize the dynamic trends of bubble changes in the form of visual charts. The deep learning approach proposed in this paper provides a new perspective for intelligent analysis of bubbles in microfluidic chip experiments, and future work aiming to its application to the complex multiphase flow field in porous media.

Novel entropy-based style transfer of the object in the content image using deep learning

Novel entropy-based style transfer of the object in the content image using deep learning

A Novel GAN Architecture Reconstructed Using Bi-LSTM and Style Transfer for PV Temporal Dynamics Simulation

A Novel GAN Architecture Reconstructed Using Bi-LSTM and Style Transfer for PV Temporal Dynamics Simulation

Domain generalization for machine compound fault diagnosis by Domain-Relevant Joint Distribution Alignment

In machine fault diagnosis, domain generalization methods have gained significant attention due to their advantage in non-requirement of priori target domain distribution. However, they pose great challenges in domain-relevant feature learning and incurs inferior unseen-domain classification when large domain divergence exists. To address this issuse, we propose a novel distribution alignment strategy named DRJDA (Domain-Relevant Joint Distribution Alignment) that matches the domain-joint distribution and domain-relevant distribution for domain generalization fault diagnosis. Specifically, the α-PE divergence is employed to minimize the distribution discrepancy, which is demonstrated to be explicitly derived as the maximum value of a quadratic function. Additionally, a parameter-free plug-and-play data augmentation module that performs feature-level instance mixture and style transfer to increase the generalization ability. Finally, data from the China Light-duty Vehicle Test Cycle (CLTC) tests are used as case studies, and the experiments carried out across 14 different domains prove the proficiency of the proposed DRJDA in learning domain-invariant feature when significant domain divergence exists, indicating its remarkable potential in compound fault diagnosis for industrial machinery.

Flow style-aware network for arbitrary style transfer

Researchers have recently proposed arbitrary style transfer methods based on various model frameworks. Although all of them have achieved good results, they still face the problems of insufficient stylization, artifacts and inadequate retention of content structure. In order to solve these problems, we propose a flow style-aware network (FSANet) for arbitrary style transfer, which combines a VGG network and a flow network. FSANet consists of a flow style transfer module (FSTM), a dynamic regulation attention module (DRAM), and a style feature interaction module (SFIM). The flow style transfer module uses the reversible residue block features of the flow network to create a sample feature containing the target content and style. To adapt the FSTM to VGG networks, we design the dynamic regulation attention module and exploit the sample features both at the channel and pixel levels. The style feature interaction module computes a style tensor that optimizes the fused features. Extensive qualitative and quantitative experiments demonstrate that our proposed FSANet can effectively avoid artifacts and enhance the preservation of content details while migrating style features.

Ancient Painting Inpainting with Regional Attention-Style Transfer and Global Context Perception

Ancient paintings, as a vital component of cultural heritage, encapsulate a profound depth of cultural significance. Over time, they often suffer from different degradation conditions, leading to damage. Existing ancient painting inpainting methods struggle with semantic discontinuities, blurred textures, and details in missing areas. To address these issues, this paper proposes a generative adversarial network (GAN)-based ancient painting inpainting method named RG-GAN. Firstly, to address the inconsistency between the styles of missing and non-missing areas, this paper proposes a Regional Attention-Style Transfer Module (RASTM) to achieve complex style transfer while maintaining the authenticity of the content. Meanwhile, a multi-scale fusion generator (MFG) is proposed to use the multi-scale residual downsampling module to reduce the size of the feature map and effectively extract and integrate the features of different scales. Secondly, a multi-scale fusion mechanism leverages the Multi-scale Cross-layer Perception Module (MCPM) to enhance feature representation of filled areas to solve the semantic incoherence of the missing region of the image. Finally, the Global Context Perception Discriminator (GCPD) is proposed for the deficiencies in capturing detailed information, which enhances the information interaction across dimensions and improves the discriminator’s ability to identify specific spatial areas and extract critical detail information. Experiments on the ancient painting and ancient Huaniao++ datasets demonstrate that our method achieves the highest PSNR values of 34.62 and 23.46 and the lowest LPIPS values of 0.0507 and 0.0938, respectively.