Contextual Semantic Information Research Articles

Metal Surface Defect Detection Based on a Transformer with Multi-Scale Mask Feature Fusion

In the production process of metal industrial products, the deficiencies and limitations of existing technologies and working conditions can have adverse effects on the quality of the final products, making surface defect detection particularly crucial. However, collecting a sufficient number of samples of defective products can be challenging. Therefore, treating surface defect detection as a semi-supervised problem is appropriate. In this paper, we propose a method based on a Transformer with pruned and merged multi-scale masked feature fusion. This method learns the semantic context from normal samples. We incorporate the Vision Transformer (ViT) into a generative adversarial network to jointly learn the generation in the high-dimensional image space and the inference in the latent space. We use an encoder–decoder neural network with long skip connections to capture information between shallow and deep layers. During training and testing, we design block masks of different scales to obtain rich semantic context information. Additionally, we introduce token merging (ToMe) into the ViT to improve the training speed of the model without affecting the training results. In this paper, we focus on the problems of rust, scratches, and other defects on the metal surface. We conduct various experiments on five metal industrial product datasets and the MVTec AD dataset to demonstrate the superiority of our method.

LM-Net: A light-weight and multi-scale network for medical image segmentation

Current medical image segmentation approaches have limitations in deeply exploring multi-scale information and effectively combining local detail textures with global contextual semantic information. This results in over-segmentation, under-segmentation, and blurred segmentation boundaries. To tackle these challenges, we explore multi-scale feature representations from different perspectives, proposing a novel, lightweight, and multi-scale architecture (LM-Net) that integrates advantages of both Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) to enhance segmentation accuracy. LM-Net employs a lightweight multi-branch module to capture multi-scale features at the same level. Furthermore, we introduce two modules to concurrently capture local detail textures and global semantics with multi-scale features at different levels: the Local Feature Transformer (LFT) and Global Feature Transformer (GFT). The LFT integrates local window self-attention to capture local detail textures, while the GFT leverages global self-attention to capture global contextual semantics. By combining these modules, our model achieves complementarity between local and global representations, alleviating the problem of blurred segmentation boundaries in medical image segmentation. To evaluate the feasibility of LM-Net, extensive experiments have been conducted on three publicly available datasets with different modalities. Our proposed model achieves state-of-the-art results, surpassing previous methods, while only requiring 4.66G FLOPs and 5.4M parameters. These state-of-the-art results on three datasets with different modalities demonstrate the effectiveness and adaptability of our proposed LM-Net for various medical image segmentation tasks.

Sentiment analysis from textual data using multiple channels deep learning models

Text sentiment analysis has been of great importance over the last few years. It is being widely used to determine a person’s feelings, opinions and emotions on any topic or for someone. In recent years, convolutional neural networks (CNNs) and long short-term memory (LSTM) have been widely adopted to develop such models. CNN has shown that it can effectively extract local information between consecutive words, but it lacks in extracting contextual semantic information between words. However, LSTM is able to extract some contextual information, where it lacks in extracting local information. To counter such problems, we applied the attention mechanism in our multi-channel CNN with bidirectional LSTM model to give attention to those parts of sentence which have major influence in determining the sentiment of that sentence. Experimental results show that our multi-channel CNN model with bidirectional LSTM and attention mechanism achieved an accuracy of 94.13% which outperforms the traditional CNN, LSTM + CNN and other machine learning algorithms.

State Transition Graph-Based Spatial–Temporal Attention Network for Cell-Level Mobile Traffic Prediction

Mobile traffic prediction enables the efficient utilization of network resources and enhances user experience. In this paper, we propose a state transition graph-based spatial–temporal attention network (STG-STAN) for cell-level mobile traffic prediction, which is designed to exploit the underlying spatial–temporal dynamic information hidden in the historical mobile traffic data. Specifically, we first identify the semantic context information over different segments of the historical data by constructing the state transition graphs, which may reveal different patterns of random fluctuation. Then, based on the state transition graphs, a spatial attention extraction module using graph convolutional networks (GCNs) is designed to aggregate the spatial information of different nodes in the state transition graph. Moreover, a temporal extraction module is employed to capture the dynamic evolution and temporal correlation of the state transition graphs over time. Such a spatial–temporal attention network can be further integrated with a parallel long short-term memory (LSTM) module to improve the accuracy of mobile traffic prediction. Extensive experiments demonstrate that the STG-STAN can better exploit the spatial–temporal information hidden in the state transition graphs, achieving superior performance compared with several baselines.

A multi-branched semantic segmentation network based on twisted information sharing pattern for medical images

BackgroundSemantic segmentation plays an indispensable role in clinical diagnosis support, intelligent surgical assistance, personalized treatment planning, and drug development, making it a core area of research in smart healthcare. However, the main challenge in medical image semantic segmentation lies in the accuracy bottleneck, primarily due to the low interactivity of feature information and the lack of deep exploration of local features during feature fusion. MethodsTo address this issue, a novel approach called Twisted Information-sharing Pattern for Multi-branched Network (TP-MNet) has been proposed. This architecture facilitates the mutual transfer of features among neighboring branches at the next level, breaking the barrier of semantic isolation and achieving the goal of semantic fusion. Additionally, performing a secondary feature mining during the transfer process effectively enhances the detection accuracy. Building upon the Twisted Pattern transmission in the encoding and decoding stages, enhanced and refined modules for feature fusion have been developed. These modules aim to capture key features of lesions by acquiring contextual semantic information in a broader context. ResultsThe experiments extensively and objectively validated the TP-MNet on 5 medical datasets and compared it with 21 other semantic segmentation models using 7 metrics. Through metric analysis, image comparisons, process examination, and ablation tests, the superiority of TP-MNet was convincingly demonstrated. Additionally, further investigations were conducted to explore the limitations of TP-MNet, thereby clarifying the practical utility of the Twisted Information-sharing Pattern. ConclusionsTP-MNet adopts the Twisted Information-sharing Pattern, leading to a substantial improvement in the semantic fusion effect and directly contributing to enhanced segmentation performance on medical images. Additionally, this semantic broadcasting mode not only underscores the importance of semantic fusion but also highlights a pivotal direction for the advancement of multi-branched architectures.

Semantic Context Information Modeling With Neural Networks in Customer Order Behavior Classification

Semantic Context Information Modeling With Neural Networks in Customer Order Behavior Classification

Multi-scale context UNet-like network with redesigned skip connections for medical image segmentation

Background and ObjectiveMedical image segmentation has garnered significant research attention in the neural network community as a fundamental requirement for developing intelligent medical assistant systems. A series of UNet-like networks with an encoder-decoder architecture have achieved remarkable success in medical image segmentation. Among these networks, UNet2+ (UNet++) and UNet3+ (UNet+++) have introduced redesigned skip connections, dense skip connections, and full-scale skip connections, respectively, surpassing the performance of the original UNet. However, UNet2+ lacks comprehensive information obtained from the entire scale, which hampers its ability to learn organ placement and boundaries. Similarly, due to the limited number of neurons in its structure, UNet3+ fails to effectively segment small objects when trained with a small number of samples. MethodIn this study, we propose UNet_sharp (UNet#), a novel network topology named after the “#” symbol, which combines dense skip connections and full-scale skip connections. In the decoder sub-network, UNet# can effectively integrate feature maps of different scales and capture fine-grained features and coarse-grained semantics from the entire scale. This approach enhances the understanding of organ and lesion positions and enables accurate boundary segmentation. We employ deep supervision for model pruning to accelerate testing and enable mobile device deployment. Additionally, we construct two classification-guided modules to reduce false positives and improve segmentation accuracy. ResultsCompared to current UNet-like networks, our proposed method achieves the highest Intersection over Union (IoU) values ((92.67±0.96)%, (92.38±1.29)%, (95.36±1.22)%, (74.01±2.03)%) and F1 scores ((91.64±1.86)%, (95.70±2.16)%, (97.34±2.76)%, (84.77±2.65)%) on the semantic segmentation tasks of nuclei, brain tumors, liver, and lung nodules, respectively. ConclusionsThe experimental results demonstrate that the reconstructed skip connections in UNet successfully incorporate multi-scale contextual semantic information. Compared to most state-of-the-art medical image segmentation models, our proposed method more accurately locates organs and lesions and precisely segments boundaries.

An efficient unsupervised approach for textile defect detection through unifying texture reconstruction and semantic localization

Deep learning has shown promise in textile defect detection, but its reliance on large high-quality labeled datasets poses challenges in real-world industrial applications. This study presents a novel unsupervised defect detection framework that effectively detects various types of texture defects with limited defect-free texture samples. The framework integrates texture and semantic information using a bilateral-branch network architecture (TSUBB-Net). Specifically, TSUBB-Net employs a weighted centering loss to cluster complex texture units, emphasizing semantic information within defect regions through a channel attention mechanism. It further fuses contextual semantic information to achieve precise defect localization. Thus, the efficient fusion method combines texture and semantic information, enabling the representation of complex texture structures and mitigating the impact of image acquisition quality on defect recognition. To evaluate the effectiveness of our proposed method, we build a unique dedicated database of textile defect image segmentation, which serves as the benchmark for textile defect detection. Experimental results demonstrate that TSUBB-Net surpasses state-of-the-art methods, exhibiting excellent performance in textile defect detection. The proposed framework holds significant potential for practical applications in the textile industry, improving defect detection capabilities.

Multi-Scale Attention Feature Enhancement Network for Single Image Dehazing.

Aiming to solve the problem of color distortion and loss of detail information in most dehazing algorithms, an end-to-end image dehazing network based on multi-scale feature enhancement is proposed. Firstly, the feature extraction enhancement module is used to capture the detailed information of hazy images and expand the receptive field. Secondly, the channel attention mechanism and pixel attention mechanism of the feature fusion enhancement module are used to dynamically adjust the weights of different channels and pixels. Thirdly, the context enhancement module is used to enhance the context semantic information, suppress redundant information, and obtain the haze density image with higher detail. Finally, our method removes haze, preserves image color, and ensures image details. The proposed method achieved a PSNR score of 33.74, SSIM scores of 0.9843 and LPIPS distance of 0.0040 on the SOTS-outdoor dataset. Compared with representative dehazing methods, it demonstrates better dehazing performance and proves the advantages of the proposed method on synthetic hazy images. Combined with dehazing experiments on real hazy images, the results show that our method can effectively improve dehazing performance while preserving more image details and achieving color fidelity.

U-MLP: MLP-based ultralight refinement network for medical image segmentation

The convolutional neural network (CNN) and Transformer play an important role in computer-aided diagnosis and intelligent medicine. However, CNN cannot obtain long-range dependence, and Transformer has shortcomings in computational complexity and a large number of parameters. Recently, compared with CNN and Transformer, the Multi-Layer Perceptron (MLP)-based medical image processing network can achieve higher accuracy with smaller computational and parametric quantities. Hence, in this work, we propose an encoder-decoder network, U-MLP, based on the ReMLP block. The ReMLP block contains an overlapping sliding window mechanism and a Multi-head Gate Self-Attention (MGSA) module, where the overlapping sliding window can extract local features of the image like convolution, then combines MGSA to fuse the information extracted from multiple dimensions to obtain more contextual semantic information. Meanwhile, to increase the generalization ability of the model, we design the Vague Region Refinement (VRRE) module, which uses the primary features generated by network inference to create local reference features, thus determining the pixel class by inferring the proximity between local features and labeled features. Extensive experimental evaluation shows U-MLP boosts the performance of segmentation. In the skin lesions, spleen, and left atrium segmentation on three benchmark datasets, our U-MLP method achieved a dice similarity coefficient of 88.27%, 97.61%, and 95.91% on the test set, respectively, outperforming 7 state-of-the-art methods.

EEG Emotion Recognition by Fusion of Multi-Scale Features.

Electroencephalogram (EEG) signals exhibit low amplitude, complex background noise, randomness, and significant inter-individual differences, which pose challenges in extracting sufficient features and can lead to information loss during the mapping process from low-dimensional feature matrices to high-dimensional ones in emotion recognition algorithms. In this paper, we propose a Multi-scale Deformable Convolutional Interacting Attention Network based on Residual Network (MDCNAResnet) for EEG-based emotion recognition. Firstly, we extract differential entropy features from different channels of EEG signals and construct a three-dimensional feature matrix based on the relative positions of electrode channels. Secondly, we utilize deformable convolution (DCN) to extract high-level abstract features by replacing standard convolution with deformable convolution, enhancing the modeling capability of the convolutional neural network for irregular targets. Then, we develop the Bottom-Up Feature Pyramid Network (BU-FPN) to extract multi-scale data features, enabling complementary information from different levels in the neural network, while optimizing the feature extraction process using Efficient Channel Attention (ECANet). Finally, we combine the MDCNAResnet with a Bidirectional Gated Recurrent Unit (BiGRU) to further capture the contextual semantic information of EEG signals. Experimental results on the DEAP dataset demonstrate the effectiveness of our approach, achieving accuracies of 98.63% and 98.89% for Valence and Arousal dimensions, respectively.

Multiscale Pixel-Level and Superpixel-Level Method for Hyperspectral Image Classification: Adaptive Attention and Parallel Multi-Hop Graph Convolution

Convolutional neural networks (CNNs) and graph convolutional networks (GCNs) have led to promising advancements in hyperspectral image (HSI) classification; however, traditional CNNs with fixed square convolution kernels are insufficiently flexible to handle irregular structures. Similarly, GCNs that employ superpixel nodes instead of pixel nodes may overlook pixel-level features; both networks tend to extract features locally and cause loss of multilayer contextual semantic information during feature extraction due to the fixed kernel. To leverage the strengths of CNNs and GCNs, we propose a multiscale pixel-level and superpixel-level (MPAS)-based HSI classification method. The network consists of two sub-networks for extracting multi-level information of HSIs: a multi-scale hybrid spectral–spatial attention convolution branch (HSSAC) and a parallel multi-hop graph convolution branch (MGCN). HSSAC comprehensively captures pixel-level features with different kernel sizes through parallel multi-scale convolution and cross-path fusion to reduce the semantic information loss caused by fixed convolution kernels during feature extraction and learns adjustable weights from the adaptive spectral–spatial attention module (SSAM) to capture pixel-level feature correlations with less computation. MGCN can systematically aggregate multi-hop contextual information to better model HSIs’ spatial background structure using the relationship between parallel multi-hop graph transformation nodes. The proposed MPAS effectively captures multi-layer contextual semantic features by leveraging pixel-level and superpixel-level spectral–spatial information, which improves the performance of the HSI classification task while ensuring computational efficiency. Extensive evaluation experiments on three real-world HSI datasets demonstrate that MPAS outperforms other state-of-the-art networks, demonstrating its superior feature learning capabilities.

Contrastive learning with feature fusion for unpaired thermal infrared image colorization

Colorizing unpaired thermal infrared images is a challenging task that existing methods struggle to perform effectively, often resulting in blurry details and unclear textures. To address these issues, we propose a novel unpaired infrared image colorization method based on contrastive learning with feature fusion. Our proposed method designs the multi-scale feature extraction module, the expansion enhancement module, and the region-aware attention module in the generator. The multi-scale feature extraction module enlarges the model receptive field, achieving more comprehensive feature extraction and fusing feature information of varying scales. The expansion enhancement module enhances the network recognition ability, capturing more semantic contextual information and image details. The region-aware attention module enables the network to obtain more global and local information, adaptively focusing on the feature of image semantic regions. Furthermore, we introduce a new composite loss function that combines adversarial loss, contrastive loss, and perceptual loss, utilizing texture information extracted from the color image to further enhance the details in the generated colorized image and improve image perception. Extensive experimental results on the KAIST dataset and the FLIR dataset show that our proposed method maximally retains tiny optical details in infrared images, such as surface textures, edge contours, and small targets, making the generated color images physically highly consistent with the Ground Truth, with rich and realistic details.

Chinese text classification by combining Chinese-BERTology-wwm and GCN.

Text classification is an important and classic application in natural language processing (NLP). Recent studies have shown that graph neural networks (GNNs) are effective in tasks with rich structural relationships and serve as effective transductive learning approaches. Text representation learning methods based on large-scale pretraining can learn implicit but rich semantic information from text. However, few studies have comprehensively utilized the contextual semantic and structural information for Chinese text classification. Moreover, the existing GNN methods for text classification did not consider the applicability of their graph construction methods to long or short texts. In this work, we propose Chinese-BERTology-wwm-GCN, a framework that combines Chinese bidirectional encoder representations from transformers (BERT) series models with whole word masking (Chinese-BERTology-wwm) and the graph convolutional network (GCN) for Chinese text classification. When building text graph, we use documents and words as nodes to construct a heterogeneous graph for the entire corpus. Specifically, we use the term frequency-inverse document frequency (TF-IDF) to construct the word-document edge weights. For long text corpora, we propose an improved pointwise mutual information (PMI*) measure for words according to their word co-occurrence distances to represent the weights of word-word edges. For short text corpora, the co-occurrence information between words is often limited. Therefore, we utilize cosine similarity to represent the word-word edge weights. During the training stage, we effectively combine the cross-entropy and hinge losses and use them to jointly train Chinese-BERTology-wwm and GCN. Experiments show that our proposed framework significantly outperforms the baselines on three Chinese benchmark datasets and achieves good performance even with few labeled training sets.

Shadow Hunter: Low-Illumination Object-Detection Algorithm

Recently, object detection, which is focused on images with normal illumination levels, has achieved great success. However, the accuracy of object detection is reduced in suboptimal environments due to the images plagued by noise and low contrast. For boosting the performance of object-detection tasks under low-illumination conditions, we propose three modules for improvement: (1) the low-level feature attention (LFA) module learns to focus on the regional feature information of the object in the low-illumination environment, highlighting important features and filtering noisy information; (2) the feature fusion neck (FFN) obtains enriched feature information by fusing the feature information of the feature map after backbone; (3) the context-spatial decoupling head (CSDH) enables the classification head to focus on contextual semantic information so that the localization head obtains richer spatial details. Extensive experiments show that our algorithm realizing end-to-end detection shows good performance in low-illumination images.

Bidirectional Attentional Interaction Networks for RGB-D salient object detection

Aiming at the issues of insufficient cross-modality feature interaction and ineffective utilization of cross-modality data in RGB-D saliency object detection (SOD) tasks, we propose a Bidirectional Attentional Interaction Network (BAINet) for RGB-D SOD, which employs an encoder-decoder structure for bidirectional interaction of cross-modality features through a dual-branch progressive fusion approach. To begin with, based on the fact that RGB and depth information streams can complement each other, the bidirectional attention interaction module accomplishes bidirectional interaction between cross-modality features by capturing complementary cues from different modality data. In order to enhance the expressiveness of the fused RGB-D features, the global feature perception module endows the features with rich multi-scale contextual semantic information by enlarging the field of perception. In addition, exploring the correlation of cross-level features is vital to achieve accurate salient inference. Specifically, We introduce a cross-level guidance aggregation module to capture inter-layer dependencies and complete the integration of cross-level features, which effectively suppresses shallow cross-modality features and refines the saliency map during decoding. To improve the model training speed, a hybrid loss function is employed to train multi-branch saliency inference maps simultaneously. Extensive experiments on five publicly available datasets clearly show that the proposed model outperforms 18 state-of-the-art methods.

Cross-modal knowledge guided model for abstractive summarization

ive summarization (AS) aims to generate more flexible and informative descriptions than extractive summarization. Nevertheless, it often distorts or fabricates facts in the original article. To address this problem, some existing approaches attempt to evaluate or verify factual consistency, or design models to reduce factual errors. However, most of the efforts either have limited effects or result in lower rouge scores while reducing factual errors. In other words, it is challenging to promote factual consistency while maintaining the informativeness of generated summaries. Inspired by the knowledge graph embedding technique, in this paper, we propose a novel cross-modal knowledge guided model (CKGM) for AS, which embeds a multimodal knowledge graph (MKG) combining image entity-relationship information and textual factual information (FI) into BERT to accomplish cross-modal information interaction and knowledge expansion. The pre-training method obtains rich contextual semantic information, while the knowledge graph supplements the textual information. In addition, an entity memory embedding algorithm is further proposed to improve information fusion efficiency and model training speed. We elaborately conducted ablation experiments and evaluated our model on the Visual Genome, FewRel, MSCOCO, and CNN/DailyMail datasets. Experimental results demonstrate that our model can significantly improve the FI consistency and informativeness of generated summaries.

A Context Semantic Auxiliary Network for Image Captioning

Image captioning is a challenging task, which generates a sentence for a given image. The earlier captioning methods mainly decode the visual features to generate caption sentences for the image. However, the visual features lack the context semantic information which is vital for generating an accurate caption sentence. To address this problem, this paper first proposes the Attention-Aware (AA) mechanism which can filter out erroneous or irrelevant context semantic information. And then, AA is utilized to constitute a Context Semantic Auxiliary Network (CSAN), which can capture the effective context semantic information to regenerate or polish the image caption. Moreover, AA can capture the visual feature information needed to generate a caption. Experimental results show that our proposed CSAN outperforms the compared image captioning methods on MS COCO “Karpathy” offline test split and the official online testing server.

Customer Sentiment Recognition in Conversation Based on Contextual Semantic and Affective Interaction Information

In the e-commerce environment, conversations between customers and businesses contain a multitude of useful information about customer sentiment. By mining that information, customer sentiment can be validly identified, which is helpful in accurately identifying customer needs and improving customer satisfaction. Contextual semantics information and inter-speaker affective interaction information are two key factors for identifying customers’ sentiments in conversation. Unfortunately, none of the existing approaches consider the two factors simultaneously. In this paper, we propose a conversational sentiment analysis method based on contextual semantic and affective interaction information. The proposed approach uses different bidirectional gated recurrent unit (BiGRU) combined with attention mechanisms to encode the contextual semantic information of different types of conversational texts. For modeling affective interactions, we use directed graph structures to portray the affective interactions between speakers and encode them with affective interaction features using graph convolutional neural networks (GCN). Finally, the two features are afused to recognize customer sentiment. The experimental results on the JDDC dataset show that our model can more accurately recognize customer sentiment than other baseline models in customer service conversation.

Edge-Guided Camouflaged Object Detection via Multi-Level Feature Integration.

Camouflaged object detection (COD) aims to segment those camouflaged objects that blend perfectly into their surroundings. Due to the low boundary contrast between camouflaged objects and their surroundings, their detection poses a significant challenge. Despite the numerous excellent camouflaged object detection methods developed in recent years, issues such as boundary refinement and multi-level feature extraction and fusion still need further exploration. In this paper, we propose a novel multi-level feature integration network (MFNet) for camouflaged object detection. Firstly, we design an edge guidance module (EGM) to improve the COD performance by providing additional boundary semantic information by combining high-level semantic information and low-level spatial details to model the edges of camouflaged objects. Additionally, we propose a multi-level feature integration module (MFIM), which leverages the fine local information of low-level features and the rich global information of high-level features in adjacent three-level features to provide a supplementary feature representation for the current-level features, effectively integrating the full context semantic information. Finally, we propose a context aggregation refinement module (CARM) to efficiently aggregate and refine the cross-level features to obtain clear prediction maps. Our extensive experiments on three benchmark datasets show that the MFNet model is an effective COD model and outperforms other state-of-the-art models in all four evaluation metrics (Sα, Eϕ, Fβw, and MAE).