Generation Module Research Articles

Considering Nonstationary within Multivariate Time Series with Variational Hierarchical Transformer for Forecasting

The forecasting of Multivariate Time Series (MTS) has long been an important but challenging task. Due to the non-stationary problem across long-distance time steps, previous studies primarily adopt stationarization method to attenuate the non-stationary problem of original series for better predictability. However, existed methods always adopt the stationarized series, which ignore the inherent non-stationarity, and have difficulty in modeling MTS with complex distributions due to the lack of stochasticity. To tackle these problems, we first develop a powerful hierarchical probabilistic generative module to consider the non-stationarity and stochastity characteristics within MTS, and then combine it with transformer for a well-defined variational generative dynamic model named Hierarchical Time series Variational Transformer (HTV-Trans), which recovers the intrinsic non-stationary information into temporal dependencies. Being an powerful probabilistic model, HTV-Trans is utilized to learn expressive representations of MTS and applied to the forecasting tasks. Extensive experiments on diverse datasets show the efficiency of HTV-Trans on MTS forecasting tasks.

Innovative design for thermoelectric power generation: Two-stage thermoelectric generator with variable twist ratio twisted tapes optimizing maximum output

In recent years, considerable effort has been dedicated to the development of highly efficient thermoelectric generators for waste heat recovery and thermoelectric power generation. In this study, we present employing twisted tapes with variable twist ratio to enhance thermal energy extraction efficiency, coupled with two-stage thermoelectric modules for heat-to-electricity conversion, resulting in a substantial increase in the power output of the thermoelectric generator. We established an experimental system that validated the superior power generation and heat recovery characteristics of the two-stage thermoelectric generator. Building upon these findings, we propose further optimizing the variable twist ratio twisted tapes to enhance the power output. We investigated the impact of tape pitch ratio, twist ratio, and twist ratio variation range on thermoelectric performance. Experimental results indicate that the influence of flow instability is more pronounced than that of swirl intensity, and twist tapes with the maximum twist ratio variation rate yield the highest net output power. Compared to an unmodified thermoelectric generator, the two-stage thermoelectric generator employing twist tapes with a twist ratio increase from π to 3π achieves a maximum net output power gain of up to 100%. These findings provide a practical framework for integrating innovative power generation modules and optimized heat exchanger designs into the application of waste heat recovery thermoelectric generators, marking a significant advancement in the field of thermoelectric generators.

TSIC-CLIP: Traffic Scene Image Captioning Model Based on Clip

Image captioning in traffic scenes presents several challenges, including imprecise caption generation, lack of personalization, and an unwieldy number of model parameters. We propose a new image captioning model for traffic scenes to address these issues. The model incorporates an adapter-based fine-tuned feature extraction part to enhance personalization and a caption generation module using global weighted attention pooling to reduce model parameters and improve accuracy. The proposed model consists of four main stages. In the first stage, the Image-Encoder extracts the global features of the input image and divides it into nine sub-regions, encoding each sub-region separately. In the second stage, the Text-Encoder encodes the text dataset to obtain text features. It then calculates the similarity between the image sub-region features and encoded text features, selecting the text features with the highest similarity. Subsequently, the pre-trained Faster RCNN model extracts local image features. The model then splices together the text features, global image features, and local image features to fuse the multimodal information. In the final stage, the extracted features are fed into the Captioning model, which effectively fuses the different features using a novel global weighted attention pooling layer. The Captioning model then generates natural language image captions. The proposed model is evaluated on the MS-COCO dataset, Flickr 30K dataset, and BUUISE-Image dataset, using mainstream evaluation metrics. Experiments demonstrate significant improvements across all evaluation metrics on the public datasets and strong performance on the BUUISE-Image traffic scene dataset.

NDOrder: Exploring a novel decoding order for scene text recognition

Text recognition in scene images is still considered as a challenging task for the computer vision and pattern recognition community. For text images affected by multiple adverse factors, such as occlusion (due to obstacles) and poor quality (due to blur and low resolution), the performance of the state-of-the-art scene text recognition methods degrades. The key reason is that the existing encoder–decoder framework follows fixed left-to-right decoding order, which lacks sufficient contextual information. In this paper, we present a novel decoding order where good-quality characters can first be decoded followed by low-quality characters, which preserves the contextual information irrespective of the aforementioned difficult scenarios. Our method, named NDOrder, extracts visual features with a ViT encoder and then decodes with the Random Order Generation module (ROG) for learning to decode with random decoding orders and the Vision-Content-Position module (VCP) for exploiting the connections among visual information, content and position. In addition, a new dataset named OLQT (Occluded and Low-Quality Text) is created by manually collecting text images that suffer from occlusion or low-quality from several standard text recognition datasets. The dataset is now available at https://github.com/djzhong1/OLQT. Experiments on OLQT and public scene text recognition benchmarks show that the proposed method achieves state-of-the-art performance.

Event type induction using latent variables with hierarchical relationship analysis

The conventional approach to event extraction requires predefined event types and their corresponding annotations to train event extractors. However, these prerequisites are often difficult to satisfy in real-world applications. To automatically induct event types, most work has been devoted to clustering event triggers, where a cluster of event triggers is represented as an event type. Some works use trigger semantics, while others use co-occurrence relationships to cluster triggers. However, the clustering results of event triggers obtained by the above work are not sufficiently detailed in describing event types, making it difficult to accurately determine the corresponding event types manually. This paper proposes an open-domain event type induction framework that automatically discovers a set of event types from a given corpus. Unlike previous work on event trigger clustering, this paper takes into consideration the hierarchical relationship of event types to partition the event trigger clusters into event mains and subtypes. The framework employs a latent variable-based neural generation module and a semantic-based clustering module, the former of which obtains event trigger clusters representing the main types of events by jointly projecting the co-occurrence and semantic information of event triggers into a latent space for event type latent variable mining, and the latter of which further divides these event trigger clusters into event subtypes based on semantic information. Finally, experiment results show that, compared with the benchmark model, the ETGen-Clus can improve event type quality scores of 6.23% and 3.11% on the two datasets, respectively.

Single-sequence protein structure prediction by integrating protein language models

Protein structure prediction has been greatly improved by deep learning in the past few years. However, the most successful methods rely on multiple sequence alignment (MSA) of the sequence homologs of the protein under prediction. In nature, a protein folds in the absence of its sequence homologs and thus, a MSA-free structure prediction method is desired. Here, we develop a single-sequence-based protein structure prediction method RaptorX-Single by integrating several protein language models and a structure generation module and then study its advantage over MSA-based methods. Our experimental results indicate that in addition to running much faster than MSA-based methods such as AlphaFold2, RaptorX-Single outperforms AlphaFold2 and other MSA-free methods in predicting the structure of antibodies (after fine-tuning on antibody data), proteins of very few sequence homologs, and single mutation effects. By comparing different protein language models, our results show that not only the scale but also the training data of protein language models will impact the performance. RaptorX-Single also compares favorably to MSA-based AlphaFold2 when the protein under prediction has a large number of sequence homologs.

Fast style transfer for ethnic patterns innovation

Minority ethnic patterns in China, reflecting diverse cultures, serve as symbols of ethnic identity and cultural heritage. However, the current artistic domain of minority ethnic patterns faces dual challenges: limited artistic styles and insufficient innovation efficiency. In addressing these issues, this study proposes a novel approach for ethnic pattern innovation, named Fast Style Transfer for Ethnic Pattern Innovation (FST-EPI), aiming to enhance innovation efficiency and diversify artistic styles. FST-EPI comprises three key modules: background generation, background fusion, and fast transfer. Firstly, the background generation module utilizes the Visual Geometry Group network (VGGNet) to extract features from images, generating a background image with the desired style while preserving the necessary stylistic attributes. Subsequently, textile patterns are fused with the stylized background image, forming a content-background fusion image. Finally, the selected stylistic patterns, along with the content-background fusion image, are input into the fast transfer module to produce the ultimate innovative pattern. Experimental results demonstrate that the generated images not only fully retain the core features and cultural elements of minority ethnic patterns but also successfully integrate the unique styles from one or more reference images. This proposed method exhibits outstanding efficiency and feasibility in a broad range of experiments related to ethnic pattern style transfer, as evidenced by thorough comparisons with state-of-the-art methods. Ultimately, the processing speed of this method achieves a real-time output at the level of seconds, reducing processing time by nearly 50% compared to traditional approaches.

A Region-Adaptive Local Perturbation-Based Method for Generating Adversarial Examples in Synthetic Aperture Radar Object Detection

In synthetic aperture radar (SAR) imaging, intelligent object detection methods are facing significant challenges in terms of model robustness and application security, which are posed by adversarial examples. The existing adversarial example generation methods for SAR object detection can be divided into two main types: global perturbation attacks and local perturbation attacks. Due to the dynamic changes and irregular spatial distribution of SAR coherent speckle backgrounds, the attack effectiveness of global perturbation attacks is significantly reduced by coherent speckle. In contrast, by focusing on the image objects, local perturbation attacks achieve targeted and effective advantages over global perturbations by minimizing interference from the SAR coherent speckle background. However, the adaptability of conventional local perturbations is limited because they employ a fixed size without considering the diverse sizes and shapes of SAR objects under various conditions. This paper presents a framework for region-adaptive local perturbations (RaLP) specifically designed for SAR object detection tasks. The framework consists of two modules. To address the issue of coherent speckle noise interference in SAR imagery, we develop a local perturbation generator (LPG) module. By filtering the original image, this module reduces the speckle features introduced during perturbation generation. It then superimposes adversarial perturbations in the form of local perturbations on areas of the object with weaker speckles, thereby reducing the mutual interference between coherent speckles and adversarial perturbation. To address the issue of insufficient adaptability in terms of the size variation in local adversarial perturbations, we propose an adaptive perturbation optimizer (APO) module. This optimizer adapts the size of the adversarial perturbations based on the size and shape of the object, effectively solving the problem of adaptive perturbation size and enhancing the universality of the attack. The experimental results show that RaLP reduces the detection accuracy of the YOLOv3 detector by 29.0%, 29.9%, and 32.3% on the SSDD, SAR-Ship, and AIR-SARShip datasets, respectively, and the model-to-model and dataset-to-dataset transferability of RaLP attacks are verified.

Co-GZSL: Feature Contrastive Optimization for Generalized Zero-Shot Learning

Generalized Zero-Shot Learning (GZSL) learns from only labeled seen classes during training but discriminates both seen and unseen classes during testing. In GZSL tasks, most of the existing methods commonly utilize visual and semantic features for training. Due to the lack of visual features for unseen classes, recent works generate real-like visual features by using semantic features. However, the synthesized features in the original feature space lack discriminative information. It is important that the synthesized visual features should be similar to the ones in the same class, but different from the other classes. One way to solve this problem is to introduce the embedding space after generating visual features. Following this situation, the embedded features from the embedding space can be inconsistent with the original semantic features. For another way, some recent methods constrain the representation by reconstructing the semantic features using the original visual features and the synthesized visual features. In this paper, we propose a hybrid GZSL model, named feature Contrastive optimization for GZSL (Co-GZSL), to reconstruct the semantic features from the embedded features, which ensures that the embedded features are close to the original semantic features indirectly by comparing reconstructed semantic features with original semantic features. In addition, to settle the problem that the synthesized features lack discrimination and semantic consistency, we introduce a Feature Contrastive Optimization Module (FCOM) and jointly utilize contrastive and semantic cycle-consistency losses in the FCOM to strengthen the intra-class compactness and the inter-class separability and to encourage the model to generate semantically consistent and discriminative visual features. By combining the generative module, the embedding module, and the FCOM, we achieve Co-GZSL. We evaluate the proposed Co-GZSL model on four benchmarks, and the experimental results indicate that our model is superior over current methods. Code is available at: https://github.com/zhanzhuxi/Co-GZSL.

Deficiency-induced absence of topological insulator behavior in antiperovskite Dirac metal oxide Sr3-x SnO: a DFT study

This study focuses on Sr3SnO (SSO), an antiperovskite Dirac metal-oxide (ADMO), that exhibits improved and enhanced physical properties when decreasing the number of Strontium (Sr) atoms in the crystal structure Sr3-xSnO. Stability and electronic properties calculations were performed using the density functional theory (DFT). Furthermore, transport and thermoelectric properties were determined by solving the Boltzmann transport equation implemented in BoltzTraP code. We report a topological insulator behavior when applying spin–orbit coupling (SOC) to pristine SSO. This behavior became absent when Sr concentration decreased. Moreover, when increasing oxygen concentration (Sr3SnO2), we reveal an improvement in the system’s stability. Electrical conductivity enhanced, for both Sr deficient SSO and for oxygen excess, reaching a value of 1.8×10201/Ω.m.s for x = 1. In addition, we show that Sr3-xSnO possesses both p-type and n-type nature, depending on the value of Sr deficiency concentration x, which makes this compound a suitable candidate for a thermoelectric generator (TEG) module.

Progressive deep snake for instance boundary extraction in medical images

Boundary extraction is meaningful in medical image analysis since it explicitly extracts the tissue/lesions boundary coordinates, which benefits the follow-up diagnosis processes. However, accurately extracting human tissue/lesions is challenging due to unclear target boundaries or pseudo boundaries, which could lead to inferior performance of existing boundary extraction and/or semantic segmentation methods. We propose a Progressive Deep Snake (PDS), which is designed with a multi-step contour evolution (MCE) and a sequential label generation module (SLGM) to explicitly extract the smooth and continuous boundary of different tissue/lesions even if they are unclear, with irregular shape, or occluded by pseudo boundaries. (1) MCE is designed to initialize and evolve the contours to the target boundaries, where a multi-step deep supervision strategy is designed to improve flexibility and accuracy. (2) SLGM is designed to generate easy-to-difficult supervision signals for the contour evolution, which provides targets of moderate difficulty in each step for better guiding the contour evolution. Experiments on a challenging verbal segment datasets show that our method achieves state-of-the-art performance (MR_AVBCE dataset: mIoU: 0.9234, mDice: 0.9598, mBoundF: 0.9241, mWcov: 0.9704). Our code is available at https://github.com/sysu19351118/PDS-pytorch.

Insights into Object Semantics: Leveraging Transformer Networks for Advanced Image Captioning.

Image captioning is a technique used to generate descriptive captions for images. Typically, it involves employing a Convolutional Neural Network (CNN) as the encoder to extract visual features, and a decoder model, often based on Recurrent Neural Networks (RNNs), to generate the captions. Recently, the encoder-decoder architecture has witnessed the widespread adoption of the self-attention mechanism. However, this approach faces certain challenges that require further research. One such challenge is that the extracted visual features do not fully exploit the available image information, primarily due to the absence of semantic concepts. This limitation restricts the ability to fully comprehend the content depicted in the image. To address this issue, we present a new image-Transformer-based model boosted with image object semantic representation. Our model incorporates semantic representation in encoder attention, enhancing visual features by integrating instance-level concepts. Additionally, we employ Transformer as the decoder in the language generation module. By doing so, we achieve improved performance in generating accurate and diverse captions. We evaluated the performance of our model on the MS-COCO and novel MACE datasets. The results illustrate that our model aligns with state-of-the-art approaches in terms of caption generation.

Multifield-Modulated Spintronic Terahertz Emitter Based on a Vanadium Dioxide Phase Transition.

The efficient generation and active modulation of terahertz (THz) waves are strongly required for the development of various THz applications such as THz imaging/spectroscopy and THz communication. In addition, due to the increasing degree of integration for the THz optoelectronic devices, miniaturizing the complex THz system into a compact unit is also important and necessary. Today, integrating the THz source with the modulator to develop a powerful, easy-to-adjust, and scalable or on-chip THz emitter is still a challenge. As a new type of THz emitter, a spintronic THz emitter has attracted a great deal of attention due to its advantages of high efficiency, ultrawide band, low cost, and easy integration. In this study, we have proposed a multifield-modulated spintronic THz emitter based on the VO2/Ni/Pt multilayer film structure with a wide band region of 0-3 THz. Because of the pronounced phase transition of the integrated VO2 layer, the fabricated THz emitter can be efficiently modulated via thermal or electric stimuli with a modulation depth of about one order of magnitude; the modulation depths under thermal stimulation and electrical stimulation were 91.8% and 97.3%, respectively. It is believed that this multifield modulated spintronic THz emitter will provide various possibilities for the integration of next-generation on-chip THz sources and THz modulators.

Task sub-type states decoding via group deep bidirectional recurrent neural network

Decoding brain states under different cognitive tasks from functional magnetic resonance imaging (fMRI) data has attracted great attention in the neuroimaging filed. However, the well-known temporal dependency in fMRI sequences has not been fully exploited in existing studies, due to the limited temporal-modeling capacity of the backbone machine learning algorithms and rigid training sample organization strategies upon which the brain decoding methods are built. To address these limitations, we propose a novel method for fine-grain brain state decoding, namely, group deep bidirectional recurrent neural network (Group-DBRNN) model. We first propose a training sample organization strategy that consists of a group-task sample generation module and a multiple-scale random fragment strategy (MRFS) module to collect training samples that contain rich task-relevant brain activity contrast (i.e., the comparison of neural activity patterns between different tasks) and maintain the temporal dependency. We then develop a novel decoding model by replacing the unidirectional RNNs that are widely used in existing brain state decoding studies with bidirectional stacked RNNs to better capture the temporal dependency, and by introducing a multi-task interaction layer (MTIL) module to effectively model the task-relevant brain activity contrast. Our experimental results on the Human Connectome Project task fMRI dataset (7 tasks consisting of 23 task sub-type states) show that the proposed model achieves an average decoding accuracy of 94.7% over the 23 fine-grain sub-type states. Meanwhile, our extensive interpretations of the intermediate features learned in the proposed model via visualizations and quantitative assessments of their discriminability and inter-subject alignment evidence that the proposed model can effectively capture the temporal dependency and task-relevant contrast.

A small sample bearing fault diagnosis method based on ConvGRU relation network

Considering that in the fault diagnosis of bearing based on relation network, using the sample mean value as the class prototype for each class is susceptible to outliers, resulting in inaccurate class prototypes, this paper proposes a convolutional gate recurrent unit (ConvGRU) relation network fault diagnosis model; firstly, the model utilizes a embedding module to extract bearing fault features, and then uses the ConvGRU as a learnable class prototype generator to generate class prototypes for each class. Secondly, a relation module is utilized to measure the similarity between class prototypes and the sample features of the query set, obtaining relation scores, and ultimately achieving fault diagnosis. In order to test the validity and advantages of the model, experimental verification and analysis were conducted on the case western storage rolling bearing dataset. The results of the experiment show that the class prototypes generated by the ConvGRU class prototype generation module have better discrimination and accuracy compared to the class prototypes generated by the relation network. In the 10-way 5-shot experiment, the accuracy of the model proposed in this paper reaches 99.60%, which increases by 6.63%, 5.10%, 4.80%, and 1.75% compared to k-nearest neighbor, convolutional neural network, prototypical network, and relation network. The method proposed in this paper helps to generate more accurate class prototypes and has a certain effect on improving the accuracy of model fault diagnosis.

Enhancing pseudo label quality for pedestrian and cyclist in weakly supervised 3D object detection

Weakly supervised 3D object detection for autonomous driving primarily focuses on cars because of their distinct rectangle boundaries and abundant instances. However, detecting categories with ambiguous rectangle boundaries and fewer instances than cars, such as pedestrians and cyclists, remains challenging with limited research. Ambiguity in rectangle boundaries presents significant difficulties in generating accurate 3D pseudo labels, while the scarcity of instances often leads to convergence issues during detector training. Pedestrians and cyclists are dense inside the 3D bounding boxes but sparse at corners and boundaries. Density is a practical clue for locating and discriminating pedestrians and cyclists in point clouds. This paper proposes a density-based 3D pseudo-label generation module (DPL-3D), addressing the challenges of ambiguous rectangle boundaries. Ambiguity rectangle boundaries will lead to poor pseudo-label quality. Therefore, By leveraging the density information of 3D points, our DPL-3D improves the accuracy and localization quality of the generated pseudo labels. It effectively segments background points, improving the estimation of pseudo labels’ location, dimension, and orientation. Few training samples always lead to local optima. Introducing multi-modal data in the detector network could enhance the constraints of objects’ features, but 2D images and 3D point clouds have a resolution gap. A motivation for dealing with the resolution gap is that neighboring regions with similar colors and textures in 2D images may exhibit spatial proximity in 3D space. Therefore, a multi-modal network driven by superpixel segmentation is introduced. This network enables effective discrimination between objects in 2D images and 3D point clouds, bridging the resolution gap and leveraging complementary features from both modalities. Experimental results on the KITTI dataset demonstrate the effectiveness of the proposed methods in addressing the challenges associated with weakly-supervised 3D object detection, particularly for categories with ambiguous rectangle boundaries and few instances.

A deep learning model of dance generation for young children based on music rhythm and beat

SummaryWith the development of technology, research related to music‐based dance generation models has been increasing. Some of the studies have applied algorithms to dance generation models, but these algorithms suffer from problems such as the inability to make an exact match between music and dance. To solve these problems, the research innovatively proposes a deep learning toddler dance generation model based on music rhythm and beat to extract music and dance features respectively. In addition, the study generates smooth dances through a generator module, improves the match between the dances and music generated by the model through a discriminator, and enhances the representativeness of audio features through a self‐encoder module. Finally, the study completes the validation of the model by comparing the loss functions and other aspects. Results demonstrate that this model has the smallest loss value of 17.58, and that the model generates a better match between dance and music for different music, with values of 8.9, 8, and 7.4. Compared with the comparison model, the study's model for generating dances for young children has better results in terms of dance generation. The study solves the problem that previous algorithms cannot make an exact match between music and dance, and has implications for fields such as cross‐modal generation and games.

Multi-Modal Attentive Prompt Learning for Few-shot Emotion Recognition in Conversations

Emotion recognition in conversations (ERC) has emerged as an important research area in Natural Language Processing and Affective Computing, focusing on accurately identifying emotions within the conversational utterance. Conventional approaches typically rely on labeled training samples for fine-tuning pre-trained language models (PLMs) to enhance classification performance. However, the limited availability of labeled data in real-world scenarios poses a significant challenge, potentially resulting in diminished model performance. In response to this challenge, we present the Multi-modal Attentive Prompt (MAP) learning framework, tailored specifically for few-shot emotion recognition in conversations. The MAP framework consists of four integral modules: multi-modal feature extraction for the sequential embedding of text, visual, and acoustic inputs; a multi-modal prompt generation module that creates six manually-designed multi-modal prompts; an attention mechanism for prompt aggregation; and an emotion inference module for emotion prediction. To evaluate our proposed model’s efficacy, we conducted extensive experiments on two widely recognized benchmark datasets, MELD and IEMOCAP. Our results demonstrate that the MAP framework outperforms state-of-the-art ERC models, yielding notable improvements of 3.5% and 0.4% in micro F1 scores. These findings highlight the MAP learning framework’s ability to effectively address the challenge of limited labeled data in emotion recognition, offering a promising strategy for improving ERC model performance.

Demonstration of a Monolithically Integrated Hybrid Device for Simultaneous Power Generation and Data Modulation

Demonstration of a Monolithically Integrated Hybrid Device for Simultaneous Power Generation and Data Modulation

Cross-modality person re-identification based on intermediate modal generation

In the context of cross-modal person re-identification, researchers often employ methods that utilize visible modality information to generate both an ‘X’ modality and a grayscale modality, enhancing the accuracy of person re-identification models. A lightweight network causes the ‘X’ modality through self-supervised learning of labels from visible images. In contrast, the grayscale modality is obtained through simple linear accumulation of the three RGB color channels from visual images. It can be observed that both the ‘X’ modality and grayscale modality are derived from visible images, which fails to establish a connection between the visible and infrared modalities. Therefore, this paper proposes an intermediate modality generation module to produce intermediate modality representations dynamically. By combining information from the visible, infrared, and intermediate modalities, the model is encouraged to capture modality-invariant features with cross-modal consistency. This enables person of the same identity to exhibit similar feature representations across different modalities, thereby mitigating the impact of distribution differences between visible and infrared modalities. Additionally, to facilitate the learning of appropriate intermediate modality representations, a distribution migration strategy is introduced. It guides the intermediate modality to maintain the correct distance from the visible and infrared modalities by optimizing the weights of the loss functions, preventing inadequate feature learning caused by an excessive focus on specific modality information. Furthermore, a mixed augmentation approach is proposed to alleviate disparities among multiple modalities further. By randomly cropping and combining regions of visible (infrared) modality images with infrared (visible) modality images, the generalization ability of the model in heterogeneous modalities is enhanced. Extensive comparative experiments are conducted on the SYSU-MM01 and RegDB datasets, yielding mAP values of 57.2% and 85.82%, respectively. The superior mAP performance on the RegDB dataset compared to most existing methods validates the effectiveness of the proposed approach.