Textual Prompts Research Articles

DreamIdentity: Enhanced Editability for Efficient Face-Identity Preserved Image Generation

While large-scale pre-trained text-to-image models can synthesize diverse and high-quality human-centric images, an intractable problem is how to preserve the face identity and follow the text prompts simultaneously for conditioned input face images and texts. Despite existing encoder-based methods achieving high efficiency and decent face similarity, the generated image often fails to follow the textual prompts. To ease this editability issue, we present DreamIdentity, to learn edit-friendly and accurate face-identity representations in the word embedding space. Specifically, we propose self-augmented editability learning to enhance the editability for projected embedding, which is achieved by constructing paired generated celebrity's face and edited celebrity images for training, aiming at transferring mature editability of off-the-shelf text-to-image models in celebrity to unseen identities. Furthermore, we design a novel dedicated face-identity encoder to learn an accurate representation of human faces, which applies multi-scale ID-aware features followed by a multi-embedding projector to generate the pseudo words in the text embedding space directly. Extensive experiments show that our method can generate more text-coherent and ID-preserved images with negligible time overhead compared to the standard text-to-image generation process.

Multi-Region Text-Driven Manipulation of Diffusion Imagery

Text-guided image manipulation has attracted significant attention recently. Prevailing techniques concentrate on image attribute editing for individual objects, however, encountering challenges when it comes to multi-object editing. The main reason is the lack of consistency constraints on the spatial layout. This work presents a multi-region guided image manipulation framework, enabling manipulation through region-level textual prompts. With MultiDiffusion as a baseline, we are dedicated to the automatic generation of a rational multi-object spatial distribution, where disparate regions are fused as a unified entity. To mitigate interference from regional fusion, we employ an off-the-shelf model (CLIP) to impose region-aware spatial guidance on multi-object manipulation. Moreover, when applied to the StableDiffusion, the presence of quality-related yet object-agnostic lengthy words hampers the manipulation. To ensure focus on meaningful object-specific words for efficient guidance and generation, we introduce a keyword selection method. Furthermore, we demonstrate a downstream application of our method for multi-region inversion, which is tailored for manipulating multiple objects in real images. Our approach, compatible with variants of Stable Diffusion models, is readily applicable for manipulating diverse objects in extensive images with high-quality generation, showing superb image control capabilities. Code is available at https://github.com/liyiming09/multi-region-guided-diffusion.

DiffTransfer: A Person Portrait Style Transfer Method Based on Stable Diffusion

With the widespread adoption of high-performance GPUs, NPUs, and other devices in recent years, the threshold for deploying large models on the edge has gradually lowered. Meanwhile, as social networks continue to thrive and virtual reality technology advances rapidly, there is an urgent demand for individuals to possess a realistic virtual identity to meet the ever-evolving social needs. In this context, portrait style transfer has attracted growing research interest. Different from the previous works that mainly follow the pipeline of generative adversarial network or variational autoencoder, this research proposes a method for person portrait style transfer based on the stable diffusion model. By pre-training underlying models, users only need to upload a small number of photos to generate corresponding LORA models for individuals. The approach involves style fusion using textual prompts and other models to modify portrait styles and generate images of individuals in different poses. Extensive visualization results validate the effectiveness of this work, which is able to generate realistic portrait images of people with different styles.

AI IMAGE GENERATOR THROUGH TEXT PROMT

In the era of digital content creation, the Text-to- Image Generator has emerged as a powerful and innovative tool that transforms textual descriptions into captivating visual representations. This website leverages advanced deep learning techniques to bridge the gap between imagination and reality, enabling users to effortlessly generate stunning images from their written words. Our Text-to-Image Generator website provides a user- friendly interface where individuals can input textual prompts, whether they are vivid descriptions, imaginative stories, or abstract concepts. Behind the scenes, state-of-the-art generative models decode the text and generate high-quality, contextually relevant images that reflect the essence of the input. In the digital landscape, the fusion of text and images has become a quintessential form of communication. Text-to-image generators serve as pivotal tools in this synergy, bridging the conceptual realm of language with the visual spectrum. This abstract delves into the conceptualization of a cutting-edge Text Prompt to Image Generator website, epitomizing the amalgamation of natural language processing and computer vision technologies. This innovative platform harnesses the power of advanced deep learning algorithms, empowering users to input textual descriptions and witness them materialize into vivid, high- resolution images. The generator employs state-of-the-art techniques such as Generative Adversarial Networks (GANs) and Transformer architectures to interpret nuanced textual cues, capturing abstract ideas and intricate details. Users can explore a diverse array of categories, from nature and architecture to fantastical realms, ensuring a versatile and captivating user experience. Key Words: "Text to Image Generator" , "AI Text to Image", "Text-Driven Image Creation" , "Generate Images from Text" , "Image Generator from Words".

Anonymizing eye-tracking stimuli with stable diffusion

Casual users nowadays can create almost arbitrary image content by providing textual prompts to generative machine-learning models. These models rapidly improve image quality with each new generation, providing means to create photos, paintings in different styles, and even videos. One feature of such models is the ability to take an image as input and adjust content according to a prompt. A visual obfuscation of content can be achieved for static images and videos by slightly changing persons, text, and other objects. The potential of this technique can be applied in eye-tracking experiments for post-hoc dissemination of analysis results and visualization. In this work, we discuss how the technique could serve to anonymize stimuli (e.g., for double-blind reviews, remove product placements, etc.) and protect the privacy of people visible in the stimuli. We further investigate how the application of this anonymization process influences visual saliency and the depiction of stimuli in visualization techniques. Our results show that slight image transformations do not drastically change the saliency of a scene but obfuscate objects and faces while keeping important image structures for context.

Utilizing stable diffusion and fine-tuning models in advertising production and logo creation: An application of text-to-image technology

This article delves into the implementation of text-to-image technology, taking advantage of stable diffusion and fine-tuning models, in the realms of advertising production and logo design. The conventional methods of production often encounter difficulties concerning cost, time constraints, and the task of locating suitable imagery. The solution suggested herein offers a more efficient and cost-effective alternative, enabling the generation of superior images and logos. The applied methodology is built around stable diffusion techniques, which employ variational autoencoders alongside diffusion models, yielding images based on textual prompts. In addition, the process is further refined by the application of fine-tuning models and adaptation processes using a Low-Rank Adaptation approach, which enhances the image generation procedure significantly. The Stable Diffusion Web User Interface offers an intuitive platform for users to navigate through various modes and settings. This strategy not only simplifies the production processes, but also decreases resource requirements, while providing ample flexibility and versatility in terms of image and logo creation. Results clearly illustrate the efficacy of the technique in producing appealing advertisements and logos. However, it is important to note some practical considerations, such as the quality of the final output and limitations inherent in text generation. Despite these potential hurdles, the use of artificial intelligence-generated content presents vast potential for transforming the advertising sector and digital content creation as a whole.

An inter-semiotic analysis of ideational meaning in text-prompted AI-generated images

Abstract This paper explores the inter-semiotic analysis of the ideational meaning in images generated by the text-to-image AI tool, Bing Image Creator. It adopts Kress and Van Leeuwen’s Grammar of Visual Design as its theoretical framework as the original grounding of the framework in systemic functional grammar (SFG) ensures a solid theoretical basis for undertaking analyses that involve the incorporation of textual and visual components. The integration of an AI generative model within the analytical framework enables a systematic connection between language and visual representations. This incorporation offers the potential to generate well-regulated pictorial representations that are systematically grounded in controlled textual prompts. This approach introduces a novel avenue for re-examining inter-semiotic processes, leveraging the power of AI technology. The paper argues that visual representations possess unique structural devices that surpass the limitations of verbal or written communication as they readily accommodate larger amounts of information in contrast to the limitations of the linear nature of alphabetic writing. Moreover, this paper extends its contribution by critically evaluating specific aspects of the Grammar of Visual Design.

Personalized Text-to-Image Model Enhancement Strategies: SOD Preprocessing and CNN Local Feature Integration

Recent advancements in text-to-image models have been substantial, generating new images based on personalized datasets. However, even within a single category, such as furniture, where the structures vary and the patterns are not uniform, the ability of the generated images to preserve the detailed information of the input images remains unsatisfactory. This study introduces a novel method to enhance the quality of the results produced by text-image models. The method utilizes mask preprocessing with an image pyramid-based salient object detection model, incorporates visual information into input prompts using concept image embeddings and a CNN local feature extractor, and includes a filtering process based on similarity measures. When using this approach, we observed both visual and quantitative improvements in CLIP text alignment and DINO metrics, suggesting that the generated images more closely follow the text prompts and more accurately reflect the input image’s details. The significance of this research lies in addressing one of the prevailing challenges in the field of personalized image generation: enhancing the capability to consistently and accurately represent the detailed characteristics of input images in the output. This method enables more realistic visualizations through textual prompts enhanced with visual information, additional local features, and unnecessary area removal using a SOD mask; it can also be beneficial in fields that prioritize the accuracy of visual data.

Language-Aware Soft Prompting: Text-to-Text Optimization for Few- and Zero-Shot Adaptation of V &L Models

AbstractSoft prompt learning has emerged as a promising direction for adapting V &L models to a downstream task using a few training examples. However, current methods significantly overfit the training data suffering from large accuracy degradation when tested on unseen classes from the same domain. In addition, all prior methods operate exclusively under the assumption that both vision and language data is present. To this end, we make the following 5 contributions: (1) To alleviate base class overfitting, we propose a novel Language-Aware Soft Prompting (LASP) learning method by means of a text-to-text cross-entropy loss that maximizes the probability of the learned prompts to be correctly classified with respect to pre-defined hand-crafted textual prompts. (2) To increase the representation capacity of the prompts, we also propose grouped LASP where each group of prompts is optimized with respect to a separate subset of textual prompts. (3) Moreover, we identify a visual-language misalignment introduced by prompt learning and LASP, and more importantly, propose a re-calibration mechanism to address it. (4) Importantly, we show that LASP is inherently amenable to including, during training, virtual classes, i.e. class names for which no visual samples are available, further increasing the robustness of the learned prompts. Expanding for the first time the setting to language-only adaptation, (5) we present a novel zero-shot variant of LASP where no visual samples at all are available for the downstream task. Through evaluations on 11 datasets, we show that our approach (a) significantly outperforms all prior works on soft prompting, and (b) matches and surpasses, for the first time, the accuracy on novel classes obtained by hand-crafted prompts and CLIP for 8 out of 11 test datasets. Finally, (c) we show that our zero-shot variant improves upon CLIP without requiring any extra data. Code will be made available.

Generating Parametric BRDFs from Natural Language Descriptions

AbstractArtistic authoring of 3D environments is a laborious enterprise that also requires skilled content creators. There have been impressive improvements in using machine learning to address different aspects of generating 3D content, such as generating meshes, arranging geometry, synthesizing textures, etc. In this paper we develop a model to generate Bidirectional Reflectance Distribution Functions (BRDFs) from descriptive textual prompts. BRDFs are four dimensional probability distributions that characterize the interaction of light with surface materials. They are either represented parametrically, or by tabulating the probability density associated with every pair of incident and outgoing angles. The former lends itself to artistic editing while the latter is used when measuring the appearance of real materials. Numerous works have focused on hypothesizing BRDF models from images of materials. We learn a mapping from textual descriptions of materials to parametric BRDFs. Our model is first trained using a semi‐supervised approach before being tuned via an unsupervised scheme. Although our model is general, in this paper we specifically generate parameters for MDL materials, conditioned on natural language descriptions, within NVIDIA's Omniverse platform. This enables use cases such as real‐time text prompts to change materials of objects in 3D environments such as “dull plastic” or “shiny iron”. Since the output of our model is a parametric BRDF, rather than an image of the material, it may be used to render materials using any shape under arbitrarily specified viewing and lighting conditions.

DILF: Differentiable rendering-based multi-view Image–Language Fusion for zero-shot 3D shape understanding

Zero-shot 3D shape understanding aims to recognize “unseen” 3D categories that are not present in training data. Recently, Contrastive Language–Image Pre-training (CLIP) has shown promising open-world performance in zero-shot 3D shape understanding tasks by information fusion among language and 3D modality. It first renders 3D objects into multiple 2D image views and then learns to understand the semantic relationships between the textual descriptions and images, enabling the model to generalize to new and unseen categories. However, existing studies in zero-shot 3D shape understanding rely on predefined rendering parameters, resulting in repetitive, redundant, and low-quality views. This limitation hinders the model’s ability to fully comprehend 3D shapes and adversely impacts the text–image fusion in a shared latent space. To this end, we propose a novel approach called Differentiable rendering-based multi-view Image–Language Fusion (DILF) for zero-shot 3D shape understanding. Specifically, DILF leverages large-scale language models (LLMs) to generate textual prompts enriched with 3D semantics and designs a differentiable renderer with learnable rendering parameters to produce representative multi-view images. These rendering parameters can be iteratively updated using a text–image fusion loss, which aids in parameters’ regression, allowing the model to determine the optimal viewpoint positions for each 3D object. Then a group-view mechanism is introduced to model interdependencies across views, enabling efficient information fusion to achieve a more comprehensive 3D shape understanding. Experimental results can demonstrate that DILF outperforms state-of-the-art methods for zero-shot 3D classification while maintaining competitive performance for standard 3D classification. The code is available at https://github.com/yuzaiyang123/DILP.

Using Artificial Intelligence to Generate Master-Quality Architectural Designs from Text Descriptions

The exceptional architecture designed by master architects is a shared treasure of humanity, which embodies their design skills and concepts not possessed by common architectural designers. To help ordinary designers improve the design quality, we propose a new artificial intelligence (AI) method for generative architectural design, which generates designs with specified styles and master architect quality through a diffusion model based on textual prompts of the design requirements. Compared to conventional methods dependent on heavy intellectual labor for innovative design and drawing, the proposed method substantially enhances the creativity and efficiency of the design process. It overcomes the problem of specified style difficulties in generating high-quality designs in traditional diffusion models. The research results indicated that: (1) the proposed method efficiently provides designers with diverse architectural designs; (2) new designs upon easily altered text prompts; (3) high scalability for designers to fine-tune it for applications in other design domains; and (4) an optimized architectural design workflow.

Machine Visions: Mapping Depictions of Machine Vision through AI Image Synthesis

This paper conducts a speculative examination of how AI image synthesisers, which generate novel imagery in response to inputted textual prompts — such as DALL-E, Midjourney, and Stable Diffusion — can be employed reflexively to investigate cultural representations of machine vision technologies. Such work can be framed methodologically as a form of ‘critical image synthesis’: the prompting of imagery that variously interrogates and makes visible the structural biases and cultural imperatives encoded within their originating architectures. In framing AI image synthesisers as an inverted form of machine vision — as generating, rather than classifying imagery through text — an opportunity is afforded to consider how they reflexively characterise themselves within their own latent spaces of representational possibility. Specifically, what kinds of imagery do these systems yield in response to prompts centring on keywords associated with machine vision technologies? And what does this reveal concerning how machine vision is represented and characterised across wider culture? This paper will empirically analyse a selection of prompted outputs from Stable Diffusion V2, treating them as a speculative mapping of contemporary visual themes and imaginaries surrounding machine vision technologies. This paper will then conclude by placing these outputs into dialogue with the author’s own creative practices involving machine vision, generating new image-text combinations that aim to provoke speculative analyses along alternative critical vectors.

A Study on Creative Nail Art Design Generation Based on Text Prompt: Focused on Image-Generating Artificial Intelligence Models, DALL-E 2 and Bing Image Creator

Nail art is an artistic activity that requires creativity and skill, reflecting individuality and taste by applying various colors and designs to fingernails. However, due to the limitations of human imagination, knowledge, and technical abilities, it can be challenging to continuously generate new ideas or create intricate and flawless designs. Recently, advances in artificial intelligence (AI) technology have led to the development of AI systems capable of generating images based on user input. These AI technologies can produce realistic, sophisticated, and creative images by interpreting textual prompts. As a result, they have found applications in various fields. In this study, we utilized image-generating AI models such as ‘DALL-E 2’ and ‘Bing Image Creator’ to analyze nail art designs generated based on textual prompts. We demonstrated that the type and content of prompts significantly influence the performance and output of image-generating AI models. By using AI technology as a supportive collaborator for nail artists, we can positively leverage its capabilities to enhance the diversity and quality of nail art designs. This will not only aid nail artists in their work and skill development but also contribute to the advancement of the nail industry by elevating the diversity and quality of nail art designs.

Generalizing Multiple Object Tracking to Unseen Domains by Introducing Natural Language Representation

Although existing multi-object tracking (MOT) algorithms have obtained competitive performance on various benchmarks, almost all of them train and validate models on the same domain. The domain generalization problem of MOT is hardly studied. To bridge this gap, we first draw the observation that the high-level information contained in natural language is domain invariant to different tracking domains. Based on this observation, we propose to introduce natural language representation into visual MOT models for boosting the domain generalization ability. However, it is infeasible to label every tracking target with a textual description. To tackle this problem, we design two modules, namely visual context prompting (VCP) and visual-language mixing (VLM). Specifically, VCP generates visual prompts based on the input frames. VLM joints the information in the generated visual prompts and the textual prompts from a pre-defined Trackbook to obtain instance-level pseudo textual description, which is domain invariant to different tracking scenes. Through training models on MOT17 and validating them on MOT20, we observe that the pseudo textual descriptions generated by our proposed modules improve the generalization performance of query-based trackers by large margins.

Few-shot Text-to-SQL Translation using Structure and Content Prompt Learning

A common problem with adopting Text-to-SQL translation in database systems is poor generalization. Specifically, when there is limited training data on new datasets, existing few-shot Text-to-SQL techniques, even with carefully designed textual prompts on pre-trained language models (PLMs), tend to be ineffective. In this paper, we present a divide-and-conquer framework to better support few-shot Text-to-SQL translation, which divides Text-to-SQL translation into two stages (or sub-tasks), such that each sub-task is simpler to be tackled. The first stage, called the structure stage, steers a PLM to generate an SQL structure (including SQL commands such as SELECT, FROM, WHERE and SQL operators such as <", ?>") with placeholders for missing identifiers. The second stage, called the content stage, guides a PLM to populate the placeholders in the generated SQL structure with concrete values (including SQL identifies such as table names, column names, and constant values). We propose a hybrid prompt strategy that combines learnable vectors and fixed vectors (i.e., word embeddings of textual prompts), such that the hybrid prompt can learn contextual information to better guide PLMs for prediction in both stages. In addition, we design keyword constrained decoding to ensure the validity of generated SQL structures, and structure guided decoding to guarantee the model to fill correct content. Extensive experiments, by comparing with ten state-of-the-art Text-to-SQL solutions at the time of writing, show that SC-Prompt significantly outperforms them in the few-shot scenario. In particular, on the widely-adopted Spider dataset, given less than 500 labeled training examples (5% of the official training set), SC-Prompt outperforms the previous SOTA methods by around 5% on accuracy.

On the Robustness of Dialogue History Representation in Conversational Question Answering: A Comprehensive Study and a New Prompt-based Method

Abstract Most work on modeling the conversation history in Conversational Question Answering (CQA) reports a single main result on a common CQA benchmark. While existing models show impressive results on CQA leaderboards, it remains unclear whether they are robust to shifts in setting (sometimes to more realistic ones), training data size (e.g., from large to small sets) and domain. In this work, we design and conduct the first large-scale robustness study of history modeling approaches for CQA. We find that high benchmark scores do not necessarily translate to strong robustness, and that various methods can perform extremely differently under different settings. Equipped with the insights from our study, we design a novel prompt-based history modeling approach and demonstrate its strong robustness across various settings. Our approach is inspired by existing methods that highlight historic answers in the passage. However, instead of highlighting by modifying the passage token embeddings, we add textual prompts directly in the passage text. Our approach is simple, easy to plug into practically any model, and highly effective, thus we recommend it as a starting point for future model developers. We also hope that our study and insights will raise awareness to the importance of robustness-focused evaluation, in addition to obtaining high leaderboard scores, leading to better CQA systems.1

Artists in the Archives

Ulrike Müller's Herstory Inventory (HI) is a collection of over one hundred works on paper by “feminist” artists who were given “drawing assignments” that began with textual prompts taken from an archival list of T-shirts that Müller discovered in the collections of the Lesbian Herstory Archives (LHA). HI has also had multiple incarnations as a staged reading/live performance, audio installation, collective art project, art exhibition, and book, and its relay across media participates in a fascination with the archive that has pervaded LGBTQ culture, resulting in a proliferation of new archives that is one manifestation of the “archival turn.” This essay focuses on how Müller's HI uses the LHA as a point of departure for a creative practice that not only opens lesbian feminist archives to new visibility and new publics but also creates a transgenerational dialogue around lesbian feminist politics and representation — both honoring and reviving its history and subjecting it to critique. HI's engagement with the LHA's lesbian feminist commitment to archival autonomy provides an interesting case history for radical archival politics, as tensions between counterarchives and archival critique get played out through the tensions between lesbian and queer feminisms. Returning to the politics of representation and visibility that have been so central and vexing in lesbian feminism, HI puts art practices in conversation with archival ones. The project approaches the archive through abstraction and drawing, both practices of representation that resist the realisms of documentary media such as film and photography, to enact a queer politics of visibility.

Compositional Prompting Video-language Models to Understand Procedure in Instructional Videos

Instructional videos are very useful for completing complex daily tasks, which naturally contain abundant clip-narration pairs. Existing works for procedure understanding are keen on pretraining various video-language models with these pairs and then fine-tuning downstream classifiers and localizers in predetermined category space. These video-language models are proficient at representing short-term actions, basic objects, and their combinations, but they are still far from understanding long-term procedures. In addition, the predetermined procedure category faces the problem of combination disaster and is inherently inapt to unseen procedures. Therefore, we propose a novel compositional prompt learning (CPL) framework to understand long-term procedures by prompting short-term video-language models and reformulating several classical procedure understanding tasks into general video-text matching problems. Specifically, the proposed CPL consists of one visual prompt and three compositional textual prompts (including the action prompt, object prompt, and procedure prompt), which could compositionally distill knowledge from short-term video-language models to facilitate long-term procedure understanding. Besides, the task reformulation enables our CPL to perform well in all zero-shot, few-shot, and fully-supervised settings. Extensive experiments on two widely-used datasets for procedure understanding demonstrate the effectiveness of the proposed approach.

A study of the evaluation metrics for generative images containing combinational creativity

AbstractIn the field of content generation by machine, the state-of-the-art text-to-image model, DALL⋅E, has advanced and diverse capacities for the combinational image generation with specific textual prompts. The images generated by DALL⋅E seem to exhibit an appreciable level of combinational creativity close to that of humans in terms of visualizing a combinational idea. Although there are several common metrics which can be applied to assess the quality of the images generated by generative models, such as IS, FID, GIQA, and CLIP, it is unclear whether these metrics are equally applicable to assessing images containing combinational creativity. In this study, we collected the generated image data from machine (DALL⋅E) and human designers, respectively. The results of group ranking in the Consensual Assessment Technique (CAT) and the Turing Test (TT) were used as the benchmarks to assess the combinational creativity. Considering the metrics’ mathematical principles and different starting points in evaluating image quality, we introduced coincident rate (CR) and average rank variation (ARV) which are two comparable spaces. An experiment to calculate the consistency of group ranking of each metric by comparing the benchmarks then was conducted. By comparing the consistency results of CR and ARV on group ranking, we summarized the applicability of the existing evaluation metrics in assessing generative images containing combinational creativity. In the four metrics, GIQA performed the closest consistency to the CAT and TT. It shows the potential as an automated assessment for images containing combinational creativity, which can be used to evaluate the images containing combinational creativity in the relevant task of design and engineering such as conceptual sketch, digital design image, and prototyping image.