Input Text Research Articles

Mesh-controllable multi-level-of-detail text-to-3D generation

Text-to-3D generation is a challenging but significant task and has gained widespread attention. Its capability to rapidly generate 3D digital assets holds huge potential application value in fields such as film, video games, and virtual reality. However, current methods often face several drawbacks, including long generation times, difficulties with the multi-face Janus problem, and issues like chaotic topology and redundant structures during mesh extraction. Additionally, the lack of control over the generated results limits their utility in downstream applications. To address these problems, we propose a novel text-to-3D framework capable of generating meshes with high fidelity and controllability. Our approach can efficiently produce meshes and textures that match the text description and the desired level of detail (LOD) by specifying input text and LOD preferences. This framework consists of two stages. In the coarse stage, 3D Gaussians are employed to accelerate generation speed, and weighted positive and negative prompts from various observation perspectives are used to address the multi-face Janus problem in the generated results. In the refinement stage, mesh vertices and faces are iteratively refined to enhance surface quality and output meshes and textures that meet specified LOD requirements. Compared to the state-of-the-art text-to-3D methods, extensive experiments demonstrate that the proposed method performs better in solving the multi-face Janus problem, enabling the rapid generation of 3D meshes with enhanced prompt adherence. Furthermore, the proposed framework can generate meshes with enhanced topology, offering controllable vertices and faces with textures featuring UV adaptation to achieve multi-level-of-detail(LODs) outputs. Specifically, the proposed method can preserve the output’s relevance to input texts during simplification, making it better suited for mesh editing and rendering efficiency. User studies also indicate that our framework receives higher evaluations compared to other methods.

Explainable Multimodal Deep Learning Model for Cyberbullying Detection (EMDL-CBD)

The increased use of social media and the internet is leading to an increase in cyberbully vulnerabilities as well as daily usage. Cyberbullying is a deliberate, aggressive behavior that can be committed by an individual or organization. It occurs when people communicate, post, and distribute damaging, false, or unfavorable content online. For individuals impacted, it results in emotional and mental health issues. Therefore, it is imperative to create automated techniques for the detection and prevention of cyberbullying. The majority of the research done on cyberbullying detection in recent years has been on text-based analysis. The two most significant media in incidents of cyberbullying are text and visual. This paper presents An Explainable Multimodal Deep Learning Model for Cyberbullying Detection include three steps The first step involves collecting datasets from different resources, which include images and their captions with binary classes (bullying and non-bullying). The second step applies two techniques of XAI: CNN+GradCam to analyze input images and produce visual explanations, and LSTM+LRP to analyze and interpret input text. The third step employs two techniques of data fusion (early and late). Final step represents the evaluation performance of the EMDL-CBD model based on a set of accuracy metrics

Linear time online algorithms for constructing linear-size suffix trie

The suffix trees are fundamental data structures for various kinds of string processing. The suffix tree of a text string T of length n has O(n) nodes and edges, and the string label of each edge is encoded by a pair of positions in T. Thus, even after the tree is built, the input string T needs to be kept stored and random access to T is still needed. The linear-size suffix tries (LSTs), proposed by Crochemore et al. [Linear-size suffix tries, TCS 638:171-178, 2016], are a “stand-alone” alternative to the suffix trees. Namely, the LST of an input text string T of length n occupies O(n) total space, and supports pattern matching and other tasks with the same efficiency as the suffix tree without the need to store the input text string T. Crochemore et al. proposed an offline algorithm which transforms the suffix tree of T into the LST of T in O(nlog⁡σ) time and O(n) space, where σ is the alphabet size. In this paper, we present two types of online algorithms which “directly” construct the LST, from right to left, and from left to right, without constructing the suffix tree as an intermediate structure. Both algorithms construct the LST incrementally when a new symbol is read, and do not access the previously read symbols. Both of the right-to-left construction algorithm and the left-to-right construction algorithm work in O(nlog⁡σ) time and O(n) space. The main feature of our algorithms is that the input text string does not need to be stored.

Adaptive Evolutionary Computing Ensemble Learning Model for Sentiment Analysis

Standard machine learning and deep learning architectures have been widely used in the field of sentiment analysis, but their performance is unsatisfactory if the input texts are short (e.g., social media posts). Specifically, the accuracy of standard machine learning methods crucially depends on the richness and completeness of the features used to represent the texts, and in the case of short messages, it is often difficult to obtain high-quality features. Conversely, methods based on deep learning can achieve better expressiveness, but these methods are computationally demanding and often suffer from over-fitting. This paper proposes a new adaptive evolutionary computational integrated learning model (AdaECELM) to overcome the problems encountered by traditional machine learning and deep learning models in sentiment analysis for short texts. AdaECELM consists of three phases: feature selection, sub classifier training, and global integration learning. First, a grid search is used for feature extraction and selection of term frequency-inverse document frequency (TF-IDF). Second, cuckoo search (CS) is introduced to optimize the combined hyperparameters in the sub-classifier support vector machine (SVM). Finally, the training set is divided into different feature subsets for sub-classifier training, and then the trained sub-classifiers are integrated and learned using the AdaBoost integrated soft voting method. Extensive experiments were conducted on six real polar sentiment analysis data sets. The results show that the AdaECELM model outperforms the traditional ML comparison methods according to evaluation metrics such as accuracy, precision, recall, and F1-score in all cases, and we report an improvement in accuracy exceeding 4.5%, the second-best competitor.

Exploring the Potential of DALL-E 2 in Pediatric Dermatology: A Critical Analysis.

Artificial intelligence (AI) is becoming increasingly explored for its potential applications in dermatology. Among various AI models, DALL-E 2 (San Francisco, CA: OpenAI), which generates de novoimages from textual inputs, has garnered significant attention for its remarkable photorealism. In our study, we aimed to analyze the performance of DALL-E 2 in the context of dermatology. The following 12 pediatric dermatological conditions common to ages <15 years were selected as tests: acne, atopic dermatitis, contact dermatitis, vitiligo, congenital melanocytic nevus, warts, molluscum contagiosum, seborrheic dermatitis, alopecia areata, infantile hemangioma, impetigo, and dermatophytosis, specifically tinea corporis. Representative morphological descriptions of each diagnosis, along with their corresponding names, were inputted into DALL-E 2 as textual prompts and subsequently compared. The accuracy of the generated images and their alignment with the intended descriptions were assessed. Among the total of 24 images reported, 18 were photorealisticand six were cartoons. More cartoons were generated when providing the model with morphological descriptions as textual inputs compared to when diagnoses were inputted. While not entirely accurate, acne stood out as the only diagnosis that was the most consistent and closest to the actual diagnosis. Both images of acne portrayed erythematous papules scattered across the face. However, DALL-E 2 resulted in poor performance for the remaining eleven diagnoses. They did not accurately represent the intended diagnoses nor align with their counterpart image. Moreover, most of the generated images featured lighter skin tones. In assessing DALL-E 2's applications in dermatology, our study highlights the need for more domain-specific and demographically inclusive training data in its algorithms to improve performance.

ZeroNLG: Aligning and Autoencoding Domains for Zero-Shot Multimodal and Multilingual Natural Language Generation.

Natural Language Generation (NLG) accepts input data in the form of images, videos, or text and generates corresponding natural language text as output. Existing NLG methods mainly adopt a supervised approach and rely heavily on coupled data-to-text pairs. However, for many targeted scenarios and for non-English languages, sufficient quantities of labeled data are often not available. As a result, it is necessary to collect and label data-text pairs for training, which is both costly and time-consuming. To relax the dependency on labeled data of downstream tasks, we propose an intuitive and effective zero-shot learning framework, ZeroNLG, which can deal with multiple NLG tasks, including image-to-text (image captioning), video-to-text (video captioning), and text-to-text (neural machine translation), across English, Chinese, German, and French within a unified framework. ZeroNLG does not require any labeled downstream pairs for training. During training, ZeroNLG (i) projects different domains (across modalities and languages) to corresponding coordinates in a shared common latent space; (ii) bridges different domains by aligning their corresponding coordinates in this space; and (iii) builds an unsupervised multilingual auto-encoder to learn to generate text by reconstructing the input text given its coordinate in shared latent space. Consequently, during inference, based on the data-to-text pipeline, ZeroNLG can generate target sentences across different languages given the coordinate of input data in the common space. Within this unified framework, given visual (imaging or video) data as input, ZeroNLG can perform zero-shot visual captioning; given textual sentences as input, ZeroNLG can perform zero-shot machine translation. We present the results of extensive experiments on twelve NLG tasks, showing that, without using any labeled downstream pairs for training, ZeroNLG generates high-quality and "believable" outputs and significantly outperforms existing zero-shot methods.

Effective type label-based synergistic representation learning for biomedical event trigger detection

BackgroundDetecting event triggers in biomedical texts, which contain domain knowledge and context-dependent terms, is more challenging than in general-domain texts. Most state-of-the-art models rely mainly on external resources such as linguistic tools and knowledge bases to improve system performance. However, they lack effective mechanisms to obtain semantic clues from label specification and sentence context. Given its success in image classification, label representation learning is a promising approach to enhancing biomedical event trigger detection models by leveraging the rich semantics of pre-defined event type labels.ResultsIn this paper, we propose the Biomedical Label-based Synergistic representation Learning (BioLSL) model, which effectively utilizes event type labels by learning their correlation with trigger words and enriches the representation contextually. The BioLSL model consists of three modules. Firstly, the Domain-specific Joint Encoding module employs a transformer-based, domain-specific pre-trained architecture to jointly encode input sentences and pre-defined event type labels. Secondly, the Label-based Synergistic Representation Learning module learns the semantic relationships between input texts and event type labels, and generates a Label-Trigger Aware Representation (LTAR) and a Label-Context Aware Representation (LCAR) for enhanced semantic representations. Finally, the Trigger Classification module makes structured predictions, where each label is predicted with respect to its neighbours. We conduct experiments on three benchmark BioNLP datasets, namely MLEE, GE09, and GE11, to evaluate our proposed BioLSL model. Results show that BioLSL has achieved state-of-the-art performance, outperforming the baseline models.ConclusionsThe proposed BioLSL model demonstrates good performance for biomedical event trigger detection without using any external resources. This suggests that label representation learning and context-aware enhancement are promising directions for improving the task. The key enhancement is that BioLSL effectively learns to construct semantic linkages between the event mentions and type labels, which provide the latent information of label-trigger and label-context relationships in biomedical texts. Moreover, additional experiments on BioLSL show that it performs exceptionally well with limited training data under the data-scarce scenarios.

The Use of Online Consultation Systems and Patient Experience of Primary Care: Cross-Sectional Analysis Using the General Practice Patient Survey.

NHS England encourages the use of online consultation (OC) systems alongside traditional access methods for patients to contact their general practice online and for practices to manage workflow. Access is a key driver of patients' primary care experience. The use of online technology and patient experience vary by sociodemographic characteristics. This study aims to assess the association between OC system use and patient experience of primary care in English general practice and how that varies by OC system model and practice sociodemographic characteristics (rurality, deprivation, age, and ethnicity). We categorized practices as "low-use" or "high-use" according to the volume of patient-initiated contacts made via the OC system. We considered practices using one of 2 OC systems with distinct designs and implementation models-shorter "free text" input with an embedded single workflow OC system (FT practices) and longer "mixed text" input with variation in implemented workflow OC system (MT practices). We used 2022 General Practice Patient Survey data to capture 4 dimensions of patient experience-overall experience, experience of making an appointment, continuity of care, and use of self-care before making an appointment. We used logistic regression at the practice level to explore the association between OC system use and patient experience, including interaction terms to assess sociodemographic variation. We included 287,194 responses from 2423 MT and 170 FT practices. The proportions of patients reporting positive experiences at MT and FT practices were similar or better than practices nationally, except at high-use MT practices. At high-use MT practices, patients were 19.8% (odds ratio [OR] 0.802, 95% CI 0.782-0.823) less likely to report a good overall experience; 24.5% (OR 0.755, 95% CI 0.738-0.773) less likely to report a good experience of making an appointment; and 18.9% (OR 0.811, 95% CI 0.792-0.83) less likely to see their preferred general practitioner; but 27.8% (OR 1.278, 95% CI 1.249-1.308) more likely to use self-care, compared with low-use MT practices. Opposite trends were seen at FT practices. Sociodemographic inequalities in patient experience were generally lower at high-use than low-use practices; for example, gaps in overall experience between practices with the most and fewest White patients decreased by 2.7 percentage points at MT practices and 6.4 percentage points at FT practices. Trends suggested greater improvements in experience for traditionally underserved groups-patients from urban and deprived areas, younger patients, and non-White patients. An OC system with shorter free text input and an integrated single workflow can enhance patient experience and reduce sociodemographic inequalities. Variation in patient experience between practices with different sociodemographic characteristics and OC systems underscores the importance of tailored design and implementation. Generalizing results across different OC systems is difficult due to variations in how they are integrated into practice workflows and communicated to patients.

EduChatbot: Implementing educational Chatbot for assisting the teaching-learning process by NLP-based hybrid heuristic adopted deep learning framework

Purpose A new Chatbot system is implemented to provide both voice-based and textual-based communication to address student queries without any delay. Initially, the input texts are gathered from the chat and then the gathered text is fed to pre-processing techniques like tokenization, stemming of words and removal of stop words. Then, the pre-processed data are given to the Natural Learning Process (NLP) for extracting the features, where the XLnet and Bidirectional Encoder Representations from Transformers (BERT) are utilized to extract the features. From these extracted features, the target-based fused feature pools are obtained. Then, the intent detection is carried out to extract the answers related to the user queries via Enhanced 1D-Convolutional Neural Networks with Long Short Term Memory (E1DCNN-LSTM) where the parameters are optimized using Position Averaging of Binary Emperor Penguin Optimizer with Colony Predation Algorithm (PA-BEPOCPA). Finally, the answers are extracted based on the intent of a particular student’s teaching materials like video, image or text. The implementation results are analyzed through different recently developed Chatbot detection models to validate the effectiveness of the newly developed model.Design/methodology/approach A smart model for the NLP is developed to help education-related institutions for an easy way of interaction between students and teachers with high prediction of accurate data for the given query. This research work aims to design a new educational Chatbot to assist the teaching-learning process with the NLP. The input data are gathered from the user through chats and given to the pre-processing stage, where tokenization, steaming of words and removal of stop words are used. The output data from the pre-processing stage is given to the feature extraction phase where XLnet and BERT are used. In this feature extraction, the optimal features are extracted using hybrid PA-BEPOCPA to maximize the correlation coefficient. The features from XLnet and features from BERT were given to target-based features fused pool to produce optimal features. Here, the best features are optimally selected using developed PA-BEPOCPA for maximizing the correlation among coefficients. The output of selected features is given to E1DCNN-LSTM for implementation of educational Chatbot with high accuracy and precision.Findings The investigation result shows that the implemented model achieves maximum accuracy of 57% more than Bidirectional long short-term memory (BiLSTM), 58% more than One Dimansional Convolutional Neural Network (1DCNN), 59% more than LSTM and 62% more than Ensemble for the given dataset.Originality/value The prediction accuracy was high in this proposed deep learning-based educational Chatbot system when compared with various baseline works.

Let’s Discover More API Relations: A Large Language Model-based AI Chain for Unsupervised API Relation Inference

APIs have intricate relations that can be described in text and represented as knowledge graphs to aid software engineering tasks. Existing relation extraction methods have limitations, such as limited API text corpus and affected by the characteristics of the input text. To address these limitations, we propose utilizing large language models (LLMs) (e.g., gpt-3.5) as a neural knowledge base for API relation inference. This approach leverages the entire Web used to pre-train LLMs as a knowledge base and is insensitive to the context and complexity of input texts. To ensure accurate inference, we design an AI chain consisting of three AI modules: API Fully Qualified Name (FQN) Parser, API Knowledge Extractor, and API Relation Decider. The accuracy of the API FQN Parser and API Relation Decider is 0.81 and 0.83, respectively. Using the generative capacity of the LLM and our approach’s inference capability, we achieve an average F1 value of 0.76 under the three datasets, significantly higher than the state-of-the-art method’s average F1 value of 0.40. Compared to the original CoT and modularized CoT methods, our AI chain design has improved the performance of API relation inference by 71% and 49%, respectively. Meanwhile, the prompt ensembling strategy enhances the performance of our approach by 32%. The API relations inferred by our method can be further organized into structured forms to provide support for other software engineering tasks.

Enhancing text-audio generation by music classification and Retrieval-Augmented Generation

Recent advancements in deep learning have propelled the development of AI systems capable of generating music that resonates with human emotions and preferences. However, current music generation models still struggle to align generated music with detailed textual descriptions and maintain consistency, especially for longer compositions. This paper presents an innovative approach to address these challenges by integrating genre classification and retrieval-augmented generation (RAG) into the music generation pipeline. We train advanced CNN architectures, including ResNet-50, GoogleNet, and VGG16, for accurate genre classification. The classifier is then incorporated into a RAG framework, where the most relevant pre-classified music piece is retrieved based on the input text query. The retrieved audio and the text description are then fed into the MUSICGEN model to generate a new music piece that inherits attributes from both inputs. We evaluate our system through a double-blind human study, comparing the outputs of the original MUSICGEN model with our RAG-enhanced model. The results demonstrate a significant improvement in the ability of the RAG-enhanced model to generate music embodying specific stylistic elements, as evidenced by higher average confidence scores from participants. Our work represents a significant step towards more personalized and context-aware AI-generated musical experiences, laying the foundation for future advancements in this exciting field.

Adversarial alignment and graph fusion via information bottleneck for multimodal emotion recognition in conversations

With the rapid development of social media and human–computer interaction, multimodal emotion recognition in conversations (MERC) tasks have begun to receive widespread research attention. The MERC task is to extract and fuse complementary semantic information from different modalities to classify the speaker’s emotion. However, the existing feature fusion methods usually directly map the features of other modalities into the same feature space for information fusion, which cannot eliminate the heterogeneity between different modalities and make the subsequent emotion class boundary learning more difficult. In addition, existing graph contrastive learning methods obtain consistent feature representations by maximizing mutual information between multiple views, which may lead to overfitting of the model. To tackle the above problem, we propose a novel Adversarial Alignment and Graph Fusion via Information Bottleneck for Multimodal Emotion Recognition in Conversations (AGF-IB) method. Firstly, we input video, audio, and text features into a multi-layer perceptron (MLP) to map them into separate feature spaces. Secondly, we build a generator and a discriminator for the three modal features, respectively, through adversarial representation to achieve information interaction between modalities and eliminate the heterogeneity among modalities. Thirdly, we introduce graph contrastive representation learning to capture intra-modal and inter-modal complementary semantic information and learn intra-class and inter-class boundary information of emotion categories. Furthermore, instead of maximizing the mutual information (MI) between multiple views, we use information bottleneck theory to minimize the MI between views. Specifically, we construct a graph structure for the three modal features respectively and perform contrastive representation learning on nodes with different emotions in the same modality and nodes with the same emotion in different modalities, to improve the feature representation ability of nodes. Finally, we use MLP to complete the emotional classification of the speaker. Extensive experiments show that AGF-IB can improve emotion recognition accuracy on IEMOCAP and MELD datasets. Furthermore, since AGF-IB is a general multimodal fusion and contrastive learning method, it can be applied to other multimodal tasks in a plug-and-play manner, e.g., humor detection.

Improving visual grounding with multi-modal interaction and auto-regressive vertex generation

We propose a concise and consistent network focusing on multi-task learning of Referring Expression Comprehension (REC) and Segmentation (RES) within Visual grounding (VG). To simplify the model architecture and achieve parameter sharing, we reconstruct the Visual grounding task as a floating-point coordinate generation problem based on both image and text inputs. Consequently, rather than separately predicting bounding boxes and pixel-level segmentation masks, we represent them uniformly as a sequence of coordinate tokens and output two corner points of bounding boxes and polygon vertices autoregressively. To improve the accuracy of point generation, we introduce a regression-based decoder. Inspired by bilinear interpolation, this decoder can directly predict precise floating-point coordinates, thus avoiding quantization errors. Additionally, we devise a Multi-Modal Interaction Fusion (M2IF) to address the imbalance between visual and language features in the model. This module focuses visual information on regions relevant to textual descriptions while suppressing the influence of irrelevant areas. Based on our model, Visual grounding is realized through a unified network structure. Experiments conducted on five benchmark datasets (RefCOCO, RefCOCO+, RefCOCOg, ReferItGame and Flickr30K Entities) demonstrate that the proposed unified network outperforms or is on par with many existing task-customized models. Codes are available at https://github.com/LFUSST/MMI-VG.

CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets

In the realm of digital creativity, our potential to craft intricate 3D worlds from imagination is often hampered by the limitations of existing digital tools, which demand extensive expertise and efforts. To narrow this disparity, we introduce CLAY, a 3D geometry and material generator designed to effortlessly transform human imagination into intricate 3D digital structures. CLAY supports classic text or image inputs as well as 3D-aware controls from diverse primitives (multi-view images, voxels, bounding boxes, point clouds, implicit representations, etc). At its core is a large-scale generative model composed of a multi-resolution Variational Autoencoder (VAE) and a minimalistic latent Diffusion Transformer (DiT), to extract rich 3D priors directly from a diverse range of 3D geometries. Specifically, it adopts neural fields to represent continuous and complete surfaces and uses a geometry generative module with pure transformer blocks in latent space. We present a progressive training scheme to train CLAY on an ultra large 3D model dataset obtained through a carefully designed processing pipeline, resulting in a 3D native geometry generator with 1.5 billion parameters. For appearance generation, CLAY sets out to produce physically-based rendering (PBR) textures by employing a multi-view material diffusion model that can generate 2K resolution textures with diffuse, roughness, and metallic modalities. We demonstrate using CLAY for a range of controllable 3D asset creations, from sketchy conceptual designs to production ready assets with intricate details. Even first time users can easily use CLAY to bring their vivid 3D imaginations to life, unleashing unlimited creativity.

Re2Pair: Increasing the Scalability of RePair by Decreasing Memory Usage.

The RePair compression algorithm produces a context-free grammar by iteratively substituting the most frequently occurring pair of consecutive symbols with a new symbol until all consecutive pairs of symbols appear only once in the compressed text. It is widely used in the settings of bioinformatics, machine learning, and information retrieval where random access to the original input text is needed. For example, in pangenomics, RePair is used for random access to a population of genomes. BigRePair improves the scalability of the original RePair algorithm by using Prefix-Free Parsing (PFP) to preprocess the text prior to building the RePair grammar. Despite the efficiency of PFP on repetitive text, there is a scalability issue with the size of the parse which causes a memory bottleneck in BigRePair. In this paper, we design and implement recursive RePair (denoted as ), which builds the RePair grammar using recursive PFP. Our novel algorithm faces the challenge of constructing the RePair grammar without direct access to the parse of text, relying solely on the dictionary of the text and the parse and dictionary of the parse of the text. We compare to BigRePair using SARS-CoV-2 haplotypes and haplotypes from the 1000 Genomes Project. We show that our method achieves over a 40% peak memory reduction and a speed up ranging between 12% to 79% compared to BigRePair when compressing the largest input texts in all experiments. is made publicly available under the GNU public license here: https://github.com/jkim210/Recursive-RePair.

A spatial scene reconstruction framework in emergency response scenario

Rapid and accurate acquisition and analysis of information is crucial for emergency management, but traditional methods have limitations such as incomplete information acquisition and slow processing speed. The natural language oriented spatial scene reconstruction method provides a new solution for emergency management, but existing generative models have limited understanding of spatial relationships and lack high-quality training samples. To address these issues, this paper proposes a novel spatial scene reconstruction framework. Specifically, the BERT based spatial information knowledge graph extraction method is used to encode the input text, label and classify the encoded text, identify spatial objects and relationships in the text, and accurately extract spatial information. Additionally, a large number of manual experiments were conducted to explore quantitative biases in human spatial cognition, and based on the obtained biases, a greedy resolution method based on cost functions was used to fine tune the layout of conflicting spatial objects and solve the conflicting spatial information in the spatial information knowledge graph. Finally, use graph convolutional neural networks to obtain scene knowledge graph embeddings that consider spatial constraints. In addition, a high-quality training sample set of “text-scene-knowledge graph” was constructed.

A Novel Hybrid Attention-Based Dilated Network for Depression Classification Model from Multimodal Data Using Improved Heuristic Approach

Automatic depression classification from multimodal input data is a challenging task. Modern methods use paralinguistic information such as audio and video signals. Using linguistic information such as speech signals and text data for depression classification is a complicated task in deep learning models. Best audio and video features are built to produce a dependable depression classification system. Textual signals related to depression classification are analyzed using text-based content data. Moreover, to increase the achievements of the depression classification system, audio, visual, and text descriptors are used. So, a deep learning-based depression classification model is developed to detect the person with depression from multimodal data. The EEG signal, Speech signal, video, and text are gathered from standard databases. Four stages of feature extraction take place. In the first stage, the features from the decomposed EEG signals are attained by the empirical mode decomposition (EMD) method, and features are extracted by means of linear and nonlinear feature extraction. In the second stage, the spectral features of the speech signals from the Mel-frequency cepstral coefficients (MFCC) are extracted. In the third stage, the facial texture features from the input video are extracted. In the fourth stage of feature extraction, the input text data are pre-processed, and from the pre-processed data, the textual features are extracted by using the Transformer Net. All four sets of features are optimally selected and combined with the optimal weights to get the weighted fused features using the enhanced mountaineering team-based optimization algorithm (EMTOA). The optimal weighted fused features are finally given to the hybrid attention-based dilated network (HADN). The HDAN is developed by combining temporal convolutional network (TCN) with bidirectional long short-term memory (Bi-LSTM). The parameters in the HDAN are optimized with the assistance of the developed EMTOA algorithm. At last, the classified output of depression is obtained from the HDAN. The efficiency of the developed deep learning HDAN is validated by comparing it with various traditional classification models.

Language-vision matching for text-to-image synthesis with context-aware GAN

Text-to-image generation (T2I) aims to produce visually compelling images while maintaining a high degree of semantic consistency with textual descriptions. Despite the impressive progress made by existing methods, there are problems with limited details of synthesized images and insufficient correlation between the provided text description and the generated images. To address these issues, we propose a Context-Aware Generative Adversarial Network (CA-GAN), which generates images aligned with the input text representations. Specifically, the Context-Aware Block (CA-Block) learns a semantic-adaptive transformation based on text style, enabling the effective fusion of text descriptions and image features for high-quality image generation with better language-vision matching. Furthermore, we propose an Attention Convolution Module (ACM) to identify greater representative traits and avoid the inability to capitalize on non-local contextual information, which enables our model to generate images that exhibit numerous detailed attributes while maintaining high-quality and semantic consistency. Thereafter, we integrate self-attention with convolution to enhance feature maps to reinforce the semantic information in the discriminator, emphasizing critical feature channels while suppressing extraneous information, ultimately yielding more detailed and richer images. The experimental results demonstrate the superiority of our method over the SOTA approaches. In addition, further studies confirm the effectiveness of the generated visual details, which exhibit a high degree of alignment with the input text descriptions. Notably, our attention mechanism showcases cooperative effects contributing to overall performance improvement. The code is available at: https://github.com/hylneu/CAGAN.

P-182 High-resolution synthetic embryo images generated by AI

Abstract Study question How effectively can a latent diffusion model generate high-fidelity embryo images tailored to the specific contextual needs of researchers, based on the user’s text input? Summary answer The latent diffusion model successfully generated high-resolution embryo images, providing a novel approach to address data scarcity in embryo imaging. What is known already The use of AI in generating synthetic data has been increasingly explored to address the scarcity of real-world datasets in various fields. Particularly in medical imaging, AI-generated synthetic data offers a potential solution to overcome the limitations imposed by data privacy concerns and ethical considerations. Previous studies have shown that AI can replicate complex patterns in data, but its application in generating embryo images remains less explored. Study design, size, duration Single static images of 5,133 Day 5 blastocysts and 2,093 Day 3 cleavages were retrospectively collected from seven in vitro fertilization clinics between June 2011 and May 2022. The images were analyzed along with relevant metadata including clinical information and embryo grades. Day 3 embryo grading was based on the number of blastomeres, evenness, and fragmentation percentage. Day 5 grading criteria included the inner cell mass, trophectoderm, and blastocyst stage evaluations. Participants/materials, setting, methods An AI model using latent diffusion was developed to generate 10,051 Day 5 and 10,088 Day 3 synthetic embryo images. The authenticity was assessed through visual Turing tests, where embryologists discerned real from synthetic images. For the evaluation, 200 real (100 Day 5, 100 Day 3) and 200 synthetic (100 Day 5, 100 Day 3) images were randomly chosen from each dataset, ensuring a comprehensive test of the generated images’ realism. Main results and the role of chance The AI model’s efficacy in generating synthetic embryo images was assessed through visual Turing tests on Day 5 and Day 3 embryos, yielding accuracies of 0.59 and 0.57, respectively. These accuracies fall within the 99% confidence interval of near-random performance (0.41 to 0.59), highlighting the challenge in distinguishing synthetic from real images. For Day 5 embryos, the sensitivity and specificity were recorded at 0.52 and 0.66, indicating a moderate challenge in identifying synthetic images and a relatively higher ease in recognizing real ones. Day 3 embryos presented a lower sensitivity of 0.41, suggesting greater difficulty in detecting synthetic images, while the specificity increased to 0.73, indicating a stronger ability to identify real images. Collectively, with an overall accuracy of 0.58, sensitivity of 0.47, and specificity of 0.70, these findings confirm the synthetic images’ remarkable realism, closely emulating actual embryo features to the extent that differentiation by experts proved challenging. This level of realism underscores the synthetic images’ potential to significantly enrich embryological research datasets, promising advancements in the field. Limitations, reasons for caution The study’s limitation lies in the generated synthetic embryos being based on three embryo grade categories, limiting feature diversity. Additionally, there was no quantitative assessment method for these images, relying instead on expert evaluation, which required deploying a large number of embryologists. Wider implications of the findings The model’s generation of highly realistic embryo images counters data scarcity in embryological research, potentially elevating AI’s utility, enriching educational content, advancing reproductive medicine, and ensuring ethical data usage. Trial registration number not applicable

Predictive Assessment of the Interior Quality of Apartments Based on Multimodal Data with Variable Visual Input

Multimodal analyses in the context of automated real estate valuation (AVM) offer the possibility of enriching the models with additional information, which benefits the accuracy of the models. However, this variety of data can overwhelm common machine learning models, which generally only process certain data modalities and only a fixed data quantity. This leads to a bottleneck in information processing, as in many cases much more information is available per observation, of which only a single selection or sample is used to train the algorithm and the remaining information is disregarded. We propose a multimodal network architecture that incorporates both textual and visual inputs and fuses their information. Furthermore, we introduce a training strategy that can take advantage of a variable number of input images for each real estate object. In our experiments, we test and compare several unimodal (baseline) models with our multimodal architecture. Our approach shows several advantages in terms of model performance over unimodal approaches. The results show the best performance for the multimodal model with a variable number of visual inputs, as well as improved prediction for the underrepresented classes of indoor quality, mitigating the effects of unbalanced data. With the presented approach, which efficiently combines and merges multiple data modalities, we have shown how such a method can be easily adapted to an AVM for the extraction of supplementary information.