High-level Semantics Research Articles

Point-MPP: Point Cloud Self-Supervised Learning From Masked Position Prediction.

Masked autoencoding has gained momentum for improving fine-tuning performance in many downstream tasks. However, it tends to focus on low-level reconstruction details, lacking high-level semantics and resulting in weak transfer capability. This article presents a novel jigsaw puzzle solver inspired by the idea that predicting the positions of disordered point cloud patches provides more semantic information, similar to how children learn by solving jigsaw puzzles. Our method adopts the mask-then-predict paradigm, erasing the positions of selected point patches rather than their contents. We first partition input point clouds into irregular patches and randomly erase the positions of some patches. Then, a Transformer-based model is used to learn high-level semantic features and regress the positions of the masked patches. This approach forces the model to focus on learning transfer-robust semantics while paying less attention to low-level details. To tie the predictions within the encoding space, we further introduce a consistency constraint on their latent representations to encourage the encoded features to contain more semantic cues. We demonstrate that a standard Transformer backbone with our pretraining scheme can capture discriminative point cloud semantic information. Furthermore, extensive experiments indicate that our method outperforms the previous best competitor across six popular downstream vision tasks, achieving new state-of-the-art performance. Codes will be available at https://git.openi.org.cn/OpenPointCloud/Point-MPP.

HiPy: Extracting High-Level Semantics from Python Code for Data Processing

Data science workloads frequently include Python code, but Python's dynamic nature makes efficient execution hard. Traditional approaches either treat Python as a black box, missing out on optimization potential, or are limited to a narrow domain. However, a deep and efficient integration of user-defined Python code into data processing systems requires extracting the semantics of the entire Python code. In this paper, we propose a novel approach for extracting the high-level semantics by transforming general Python functions into program generators that generate a statically-typed IR when executed. The extracted IR then allows for high-level, domain-specific optimizations and the generation of efficient C++ code. With our prototype implementation, HiPy, we achieve single-threaded speedups of 2-20x for many workloads. Furthermore, HiPy is also capable of accelerating Python code in other domains like numerical data, where it can sometimes even outperform specialized compilers.

Verified Lock-Free Session Channels with Linking

Type systems and program logics based on session types provide powerful high-level reasoning principles for message-passing concurrency. Modern versions employ bidirectional session channels that (1) are asynchronous so that send operations do not block, (2) have buffers in both directions so that both parties can send messages in parallel, and (3) feature a link operation (also called forward) to concisely write programs in process style. These features complicate a low-level lock-free implementation of channels and therefore increase the gap between the meta theory of prior work—which is verified w.r.t. a high-level semantics of channels (e.g., π-calculus)—and the code that runs on an actual computer. We address this problem by verifying a low-level lock-free implementation of session channels w.r.t. a high-level specification based on session types. We carry out our verification in a layered manner by employing the Iris framework for concurrent separation logic. We start with an abstract specification of (unidirectional) queues—of which we provide a linked-list and array-segment based implementation—and gradually build up to session channels with all of the aforementioned features. To make a layered verification possible we develop two logical abstractions—queues with ghost linking and pairing invariants—to reason about the atomicity and changing endpoints due to linking, respectively. All our results are mechanized in the Coq proof assistant.

SIMTSeg: A self-supervised multivariate time series segmentation method with periodic subspace projection and reverse diffusion for industrial process

Subsequences with varied regimes in the industrial multivariate time series (MTS) are closely associated with the dynamic status of the multi-phased industrial process. Time series segmentation (TSS) provides insights into the underlying behavior of industrial systems. However, the complexity of industrial data poses significant challenges to the conventional TSS methods. Motivated by this, a Self-supervised Industrial Multivariate Time-series Segmentation method (SIMTSeg) is presented in this work. An MTS folding module based on Ramanujan periodic subspace projection is first proposed, where the MTS is reshaped into the 3D feature map to realize the compact representation of the intricate data dependencies. Subsequently, a self-supervised module based on the encoder-decoder architecture is adopted to address the problem of deficient and task-specific annotations in industrial data. The folded feature map is denoised step by step following the reverse diffusion process, and finally turns into the segmentation mask without redundant details. The proposed SIMTSeg has been validated by a popular industrial benchmark, the Tennessee Eastman Process, and outperforms the unsupervised data-driven baselines in terms of various performance metrics. SIMTSeg has no prerequisite on the number of segmentation points or regime types, and is capable of giving more meaningful segmentation results that are in line with the high-level semantics.

Weakly supervised semantic segmentation by knowledge graph inference

The weakly supervised semantic segmentation (WSSS) training based on image-level labels in convolutional neural network (CNN) is usually divided into two stages: multi-label classification and semantic segmentation. However, most of the existing work focuses on the improvement of the multi-label classification network stage, and little effort has been done to improve the performance of the downstream segmentation networks. In addition, CNN-based local convolution lacks in modeling extensive dependencies among categories. Therefore, in this paper, we propose a graph reasoning method to improve both the upstream and the downstream stages of the multi-label classification network and the semantic segmentation networks. In the multi-label classification network, we utilize external knowledge combined with a graph convolutional network (GCN) to perform global reasoning on the dependencies of each category. In the segmentation network, the Graph Reasoning Mapping Module (GRM) is proposed to explore the knowledge acquired from text corpora and facilitate contextual reasoning in various categories of image regions. The proposed GRM module is able to enhance the feature representation of local convolutions on the high-level semantics of the segmentation network, and adaptively learn the semantic consistency of each sample. We achieve state-of-the-art WSSS performance on PASCAL VOC 2012 and MS-COCO 2014 datasets with only image-level supervision. Extensive experiments on multi-label classification networks and semantic segmentation networks demonstrate the effectiveness of our proposed graph reasoning method on WSSS. Our code is available at:https://github.com/JIA-ZHANG666/GRM_layer.

Unsupervised Illumination Adaptation for Low-Light Vision.

Insufficient lighting poses challenges to both human and machine visual analytics. While existing low-light enhancement methods prioritize human visual perception, they often neglect machine vision and high-level semantics. In this paper, we make pioneering efforts to build an illumination enhancement model for high-level vision. Drawing inspiration from camera response functions, our model could enhance images from the machine vision perspective despite being lightweight in architecture and simple in formulation. We also introduce two approaches that leverage knowledge from base enhancement curves and self-supervised pretext tasks to train for different downstream normal-to-low-light adaptation scenarios. Our proposed framework overcomes the limitations of existing algorithms without requiring access to labeled data in low-light conditions. It facilitates more effective illumination restoration and feature alignment, significantly improving the performance of downstream tasks in a plug-and-play manner. This research advances the field of low-light machine analytics and broadly applies to various high-level vision tasks, including classification, face detection, optical flow estimation, and video action recognition.

ASF-LKUNet: Adjacent-scale fusion U-Net with large kernel for multi-organ segmentation

In the multi-organ segmentation task of medical images, there are some challenging issues such as the complex background, blurred boundaries between organs, and the larger scale difference in volume. Due to the local receptive fields of conventional convolution operations, it is difficult to obtain desirable results by directly using them for multi-organ segmentation. While Transformer-based models have global information, there is a significant dependency on hardware because of the high computational demands. Meanwhile, the depthwise convolution with large kernel can capture global information and have less computational requirements. Therefore, to leverage the large receptive field and reduce model complexity, we propose a novel CNN-based approach, namely adjacent-scale fusion U-Net with large kernel (ASF-LKUNet) for multi-organ segmentation. We utilize a u-shaped encoder–decoder as the base architecture of ASF-LKUNet. In the encoder path, we design the large kernel residual block, which combines the large and small kernels and can simultaneously capture the global and local features. Furthermore, for the first time, we propose an adjacent-scale fusion and large kernel GRN channel attention that incorporates the low-level details with the high-level semantics by the adjacent-scale feature and then adaptively focuses on the more global and meaningful channel information. Extensive experiments and interpretability analysis are made on the Synapse multi-organ dataset (Synapse) and the ACDC cardiac multi-structure dataset (ACDC). Our proposed ASF-LKUNet achieves 88.41% and 89.45% DSC scores on the Synapse and ACDC datasets, respectively, with 17.96M parameters and 29.14 GFLOPs. These results show that our method achieves superior performance with favorable lower complexity against ten competing approaches.ASF-LKUNet is superior to various competing methods and has less model complexity. Code and the trained models have been released on GitHub.

Cross-stage feature fusion and efficient self-attention for salient object detection

Salient Object Detection (SOD) approaches usually aggregate high-level semantics with object details layer by layer through a pyramid fusion structure. However, the progressive feature fusion mechanism may lead to gradually dilution of valuable semantics and prediction accuracy. In this work, we propose a Cross-stage Feature Fusion Network (CFFNet) for salient object detection. CFFNet consists of a Cross-stage Semantic Fusion Module (CSF), a Feature Filtering and Fusion Module (FFM), and a progressive decoder to tackle the above problems. Specifically, to alleviate the semantics dilution problem, CSF concatenates different stage backbone features and extracts multi-scale global semantics using transformer blocks. Global semantics are then distributed to corresponding backbone stages for cross-stage semantic fusion. The FFM module implements efficient self-attention-based feature fusion. Different from regular self-attention which has quadratic computational complexity. Finally, a progressive decoder is adopted to refine saliency maps. Experimental results demonstrate that CFFNet outperforms state-of-the-arts on six SOD datasets.

UNet-like network fused swin transformer and CNN for semantic image synthesis

Semantic image synthesis approaches has been dominated by the modelling of Convolutional Neural Networks (CNN). Due to the limitations of local perception, their performance improvement seems to have plateaued in recent years. To tackle this issue, we propose the SC-UNet model, which is a UNet-like network fused Swin Transformer and CNN for semantic image synthesis. Photorealistic image synthesis conditional on the given semantic layout depends on the high-level semantics and the low-level positions. To improve the synthesis performance, we design a novel conditional residual fusion module for the model decoder to efficiently fuse the hierarchical feature maps extracted at different scales. Moreover, this module combines the opposition-based learning mechanism and the weight assignment mechanism for enhancing and attending the semantic information. Compared to pure CNN-based models, our SC-UNet combines the local and global perceptions to better extract high- and low-level features and better fuse multi-scale features. We have conducted an extensive amount of comparison experiments, both in quantitative and qualitative terms, to validate the effectiveness of our proposed SC-UNet model for semantic image synthesis. The outcomes illustrate that SC-UNet distinctively outperforms the state-of-the-art model on three benchmark datasets (Citysacpes, ADE20K, and COCO-Stuff) including numerous real-scene images.

Edge-enhanced infrared image super-resolution reconstruction model under transformer

Infrared images have important applications in military, security and surveillance fields. However, limited by technical factors, the resolution of infrared images is generally low, which seriously limits the application and development of infrared images in various fields. To address the problem of difficult recovery of edge information and easy ringing effect in the super-resolution reconstruction process of infrared images, an edge-enhanced infrared image super-resolution reconstruction model TESR under transformer is proposed. The main structure of this model is transformer. First, in view of the problem of difficult recovery of edge information of infrared images, an edge detection auxiliary network is designed, which can obtain more accurate edge information from the input low-resolution images and enhance the edge details during image reconstruction; then, the CSWin Transformer is introduced to compute the self-attention of horizontal and vertical stripes in parallel, so as to increase the receptive field of the model and enable it to utilize features with higher semantic levels. The super-resolution reconstruction model proposed in this paper can extract more comprehensive image information, and at the same time, it can obtain more accurate edge information to enhance the texture details of super-resolution images, and achieve better reconstruction results.

Selective policy transfer in multi-agent systems with sparse interactions

Previously trained single-agent strategies, which are considerably easier to acquire than multi-agent strategies, are instructive for multi-agent reinforcement learning, especially when the interactions between agents are sparse. Traditional methods of knowledge transfer, from single-agent source tasks to multi-agent target tasks, typically rely on a pre-designed Markov-decision-process similarity function or metric function for evaluating the task similarity. In this study, we propose a selective policy transfer (SPOT) algorithm that eliminates the need for the manual crafting of a metric function to assess the similarity between source and target tasks. The SPOT algorithm enables agents to autonomously determine when and which policy to transfer using well-trained single-agent policies as options in the training process. We introduced a multi-agent policy-learning option in the option library, thus allowing the SPOT algorithm to leverage the transferred knowledge while concurrently learning new policies. The SPOT algorithm efficiently transfers sequential strategies in a few steps, thereby capturing high-level semantics. Experimental results obtained in both multi-agent arcade and multi-agent particle environments have demonstrated that the proposed algorithm outperforms the state-of-the-art methods in terms of jump-start and convergence speeds. Our evaluation indicators included the jump start, steps to threshold, cumulative reward, and asymptotic performance. Furthermore, visualizations of the strategies and termination functions of the agents aid in elucidating the operational principles of the SPOT algorithm.

SIIF: Semantic information interactive fusion network for photovoltaic defect segmentation

Any defects or damage on photovoltaic panels significantly reduce photoelectric conversion efficiency and service life. Therefore, the refined defect segmentation technology is the key to the quality inspection of photovoltaic equipment. However, the existing methods ignore the information interaction between multi-levels, causing semantic features to be easily lost and difficult to fuse. In this paper, we propose a segmentation network based on an interactive fusion of semantic information for solar cell defect identification, which can effectively interact with multi-level rich information and enhance the positioning capabilities of high-level semantics. Specifically, we propose a novel Multi-Level Pyramid Strategy (MLPS) to reduce the information gap between multi-level features. MLPS adopts a pooling distribution mechanism to enable multi-level features to interact effectively in the feature pyramid. In addition, to effectively integrate high-level positioning semantics into the neighbor level, we design a new Neighbor Attention Fusion module (NAF) to propagate high-level semantic information gradually. NAF learns spatial neighbor-level similarity combined with high-level attention to balance location information and contextual features. We perform comprehensive experiments to demonstrate the advancement of our approach. Our method achieves 84.3% MIOU on our solar cell dataset, outperforming existing state-of-the-art (SOTA) methods. To verify the generalization ability of our network, we train and evaluate the PSCDE dataset. The results show that our model achieves an excellent 98.35%MIOU.

Semantic aware-based instruction embedding for binary code similarity detection.

Binary code similarity detection plays a crucial role in various applications within binary security, including vulnerability detection, malicious software analysis, etc. However, existing methods suffer from limited differentiation in binary embedding representations across different compilation environments, lacking dynamic high-level semantics. Moreover, current approaches often neglect multi-level semantic feature extraction, thereby failing to acquire precise semantic information about the binary code. To address these limitations, this paper introduces a novel detection solution called BinBcla. This method employs an enhanced pre-training model to generate instruction embeddings with dynamic semantics for binary functions. Subsequently, multi-feature fusion technique is utilized to extract local semantic information and long-distance global features from the code, respectively, employing self-attention to comprehend the structure information of the code. Finally, an improved cosine similarity method is employed to learn relationships among all elements of the distance vectors, thereby enhancing the model's robustness to new sample functions. Experiments are conducted across different architectures, compilers, and optimization levels. The results indicate that BinBcla achieves higher accuracy, precision and F1 score compared to existing methods.

Recursive Deformable Pyramid Network for Unsupervised Medical Image Registration.

Complicated deformation problems are frequently encountered in medical image registration tasks. Although various advanced registration models have been proposed, accurate and efficient deformable registration remains challenging, especially for handling the large volumetric deformations. To this end, we propose a novel recursive deformable pyramid (RDP) network for unsupervised non-rigid registration. Our network is a pure convolutional pyramid, which fully utilizes the advantages of the pyramid structure itself, but does not rely on any high-weight attentions or transformers. In particular, our network leverages a step-by-step recursion strategy with the integration of high-level semantics to predict the deformation field from coarse to fine, while ensuring the rationality of the deformation field. Meanwhile, due to the recursive pyramid strategy, our network can effectively attain deformable registration without separate affine pre-alignment. We compare the RDP network with several existing registration methods on three public brain magnetic resonance imaging (MRI) datasets, including LPBA, Mindboggle and IXI. Experimental results demonstrate our network consistently outcompetes state of the art with respect to the metrics of Dice score, average symmetric surface distance, Hausdorff distance, and Jacobian. Even for the data without the affine pre-alignment, our network maintains satisfactory performance on compensating for the large deformation. The code is publicly available at https://github.com/ZAX130/RDP.

ADNet: A Real-Time Floating Algae Segmentation Using Distillation Network

The segmentation of floating algae is a hot topic in the field of marine environmental research. Given the vastness of coastal areas and complex environments, algae detection models must have both higher performance and lower deployment costs. However, relying solely on a single Convolutional Neural Network (CNN) or transformer structure fails to achieve this objective. In this paper, a novel real-time floating algae segmentation method using a distillation network (ADNet) is proposed, based on the RGB images. ADNet can effectively transfer the performance of the transformer-based teacher network to the CNN-based student model while preserving its lightweight design. Faced with complex marine environments, we introduce a novel Channel Purification Module (CPM) to simultaneously strengthen algae features and purify interference responses. Importantly, the CPM achieves this operation without increasing any learnable parameters. Moreover, considering the huge scale differences among algae targets in surveillance RGB images, we propose a lightweight multi-scale feature fusion network (L-MsFFN) to improve the student’s modeling ability across various scales. Additionally, to mitigate interference from low-level noises on higher-level semantics, a novel position purification module (PPM) is proposed. The PPM can achieve more accurate weight attention calculation between different pyramid levels, thereby enhancing the effectiveness of fusion. Compared to CNNs and transformers, our ADNet strikes an optimal balance between performance and speed. Extensive experimental results demonstrate that our ADNet achieves higher application performance in the field of floating algae monitoring tasks.

Optimizing intrinsic representation for tracking

Learning target representations dedicated to tracking tasks is a promising direction in visual object tracking. Current state-of-the-art approaches exploit pixel appearance reconstruction of auto-encoders for representation training. Such pixel ways are redundant for the high-level semantics of tracking and suffer from an optimization gap where better appearance reconstruction does not correspond to better tracking performance. This article explores the problem of reconstruction misalignment of target representations in Siamese network architectures. We propose an optimization framework for intrinsic representation that restores the high-level feature loss between the networks on the reconstruction side and the target side. This intrinsic representation is generated by feeding the masked image tokens to the target side branch. Unlike the pixel ways, our approach learns to reconstruct the semantic latent level of representation for tracking objects. Moreover, we investigate feeding strategies from the information bottleneck perspective for masked patches to increase the masking performance yet reduce the computational complexity. Additionally, robust propagation mechanisms for the weight parameters of two-sided networks are also explored. Comprehensive evaluations on seven tracking benchmarks demonstrate our effectiveness: GOT-10k, LaSOT, TrackingNet, NFS, UAV123, TNL2K, and OTB100. Compared to the previous best pixel tracker, we outperform it on all seven datasets, with a maximum absolute score of 5.9% on average overlap and a Graphics Processing Unit (GPU) speed of 109 frames per second.

Customizing graph neural networks using path reweighting

Graph Neural Networks (GNNs) have been extensively used for mining graph-structured data with impressive performance. However, because these traditional GNNs do not distinguish among various downstream tasks, embeddings embedded by them are not always effective. Intuitively, paths in a graph imply different semantics for different downstream tasks. Inspired by this, we design a novel GNN solution, namely Customized Graph Neural Network with Path Reweighting (CustomGNN for short). Specifically, the proposed CustomGNN can automatically learn the high-level semantics for specific downstream tasks to highlight semantically relevant paths as well to filter out task-irrelevant noises in a graph. Furthermore, we empirically analyze the semantics learned by CustomGNN and demonstrate its ability to avoid the three inherent problems in traditional GNNs, i.e., over-smoothing, poor robustness, and overfitting. In experiments with the node classification task, CustomGNN achieves state-of-the-art accuracies on three standard graph datasets and four large graph datasets. The source code of the proposed CustomGNN is available at https://github.com/cjpcool/CustomGNN.

Transformer-CNN hybrid network for crowd counting

Efficient feature representation is the key to improving crowd counting performance. CNN and Transformer are the two commonly used feature extraction frameworks in the field of crowd counting. CNN excels at hierarchically extracting local features to obtain a multi-scale feature representation of the image, but it struggles with capturing global features. Transformer, on the other hand, could capture global feature representation by utilizing cascaded self-attention to capture remote dependency relationships, but it often overlooks local detail information. Therefore, relying solely on CNN or Transformer for crowd counting has certain limitations. In this paper, we propose the TCHNet crowd counting model by combining the CNN and Transformer frameworks. The model employs the CMT (CNNs Meet Vision Transformers) backbone network as the Feature Extraction Module (FEM) to hierarchically extract local and global features of the crowd using a combination of convolution and self-attention mechanisms. To obtain more comprehensive spatial local information, an improved Progressive Multi-scale Learning Process (PMLP) is introduced into the FEM, guiding the network to learn at different granularity levels. The features from these three different granularity levels are then fed into the Multi-scale Feature Aggregation Module (MFAM) for fusion. Finally, a Multi-Scale Regression Module (MSRM) is designed to handle the multi-scale fused features, resulting in crowd features rich in high-level semantics and low-level detail. Experimental results on five benchmark datasets demonstrate that TCHNet achieves highly competitive performance compared to some popular crowd counting methods.

Graph4IUR: Incomplete Utterance Rewriting with Semantic Graph

Utterance rewriting aims to identify and supply the omitted information in human conversation, which further enables the downstream task to understand conversations more comprehensively. Recently, sequence edit methods, which leverage the overlap between two sentences, have been widely applied to narrow the search space confronted by the previous linear generation methods. However, these methods ignore the relationship between linguistic elements in the conversation, which reflects how the knowledge and thoughts are organized in human communication. In this case, although most of the content in rewritten sentences can be found in the context, we found that some connecting words expressing relationships are often missing, which results in the out-of-context problem for the previous sentence edit method. To that end, in this paper, we propose a new semantic Graph-based Incomplete Utterance Rewriting (Graph4IUR) framework, which takes the semantic graph to depict the relationship between linguistic elements and captures out-of-context words. Specifically, we adopt the Abstract Meaning Representation (AMR) [4] graph as the basic sentence-to-graph method to depict the dialogue from the graph perspective, which could well represent the high-level semantics relationships of sentences. Along this line, we further adapt the sentence editing models to rewrite without changing the sentence architecture, which brings a restriction to exploring the overlap part of the current and rewritten sentences in the IUR task. Extensive experimental results indicate that our Graph4IUR framework can effectively alleviate the out-of-context problem and improve the performance of the previous edit-based methods in the IUR task.

聚焦可变形注意力的马铃薯芽苗精确无损检测方法

Accurate potato sprout detection is the key to automatic seed potato cutting, which is important for potato quality and yield. In this paper, a lightweight DAS-YOLOv8 model is proposed for the potato sprout detection task. By embedding DAS deformable attention in the feature extraction network and the feature fusion network, the global feature context can be efficiently represented and the attention increased to the relevant pixel image region; then, the C2f_Atten module fusing Shuffle attention is designed based on the C2f module to satisfy the attention to the key feature information of the high-level abstract semantics of the feature extraction network. At the same time, the ghost convolution is introduced to improve the C2f module and convolutional module to realize the decomposition of the redundant features to extract the key features. Verified on the collected potato sprout image data set, the average accuracy of the proposed DAS-YOLOv8 model is 94.25%, and the calculation amount is only 7.66 G. Compared with the YOLOv8n model, the accuracy is 2.13% higher, and the average accuracy is 1.55% higher. In comparison to advanced state-of-the-art (SOTA) target detection algorithms, the method in this paper offers a better balance between comprehensive performance and lightweight model design. The improved and optimized DAS-YOLOv8 model can realize the effective detection of potato sprouts, meet the requirements of real-time processing, and can provide theoretical support for the non-destructive detection of sprouts in automatic seed potato cutting.