Semantic Completion Research Articles

H2GFormer: Horizontal-to-Global Voxel Transformer for 3D Semantic Scene Completion

3D Semantic Scene Completion (SSC) has emerged as a novel task in vision-based holistic 3D scene understanding. Its objective is to densely predict the occupancy and category of each voxel in a 3D scene based on input from either LiDAR or images. Currently, many transformer-based semantic scene completion frameworks employ simple yet popular Cross-Attention and Self-Attention mechanisms to integrate and infer dense geometric and semantic information of voxels. However, they overlook the distinctions among voxels in the scene, especially in outdoor scenarios where the horizontal direction contains more variations. And voxels located at object boundaries and within the interior of objects exhibit varying levels of positional significance. To address this issue, we propose a transformer-based SSC framework called H2GFormer that incorporates a horizontal-to-global approach. This framework takes into full consideration the variations of voxels in the horizontal direction and the characteristics of voxels on object boundaries. We introduce a horizontal window-to-global attention (W2G) module that effectively fuses semantic information by first diffusing it horizontally from reliably visible voxels and then propagating the semantic understanding to global voxels, ensuring a more reliable fusion of semantic-aware features. Moreover, an Internal-External Position Awareness Loss (IoE-PALoss) is utilized during network training to emphasize the critical positions within the transition regions between objects. The experiments conducted on the SemanticKITTI dataset demonstrate that H2GFormer exhibits superior performance in both geometric and semantic completion tasks. Our code is available on https://github.com/Ryanwy1/H2GFormer.

A patent text-based product conceptual design decision-making approach considering the fusion of incomplete evaluation semantic and scheme beliefs

Conceptual design evaluation is a significant challenge in product development to select suitable conceptual schemes (CS). Relying on decision makers' (DMs) personal experience for design information may result in incomplete and ambiguous evaluation semantics, creating an incomplete decision matrix. Previous approaches use attributes based on candidate schemes to complete the missing information, but neglect the mining of associated design information from external resources such as patents, reducing the beliefs objectivity after missing semantic completion. Besides, non-compensatory nature between DMs’ interactivity and criteria have not been well considered. To fill these issues, a patent text-based CS decision approach considering the fusion of incomplete semantics and scheme beliefs is proposed. First, using the beneficial effects text from patents as historical design data, constructing general evaluation criteria using the Latent Dirichlet Allocation topic model, and introducing the Apriori model to extract strong association rules between criteria to complete the incomplete semantics. Second, a fuzzy measure model for DMs is constructed based on intuitionistic fuzzy cross-entropy and Shapley value method, supporting the allocation of DMs' interactive weights. Then, the fuzzy Dempster-Shafer evidence theory is used to transform the multi-criteria decision model into a belief fusion problem of CS that avoids aggregating the evaluation semantics of non-compensatory criteria and outputting the optimal CS with interval credibility. A practical case study of an in-pipe inspection robot will be employed to validate the proposed approach, sensitivity analysis and comparison results confirmed that the complement data have the reliability to avoid continuous investment in the evaluation process.

Dense Top-View Semantic Completion With Sparse Guidance and Online Distillation

Dense semantic scene understanding of the surrounding environment in top-view is a crucial task for autonomous vehicles. Recent LiDAR-based semantic perception works mainly focus on point-wise predictions of the LiDAR points instead of dense predictions of the environment, making them not appropriate for path-planning tasks. Pillar and voxel representations can achieve dense predictions, but the generation of data representation and data processing are usually time-consuming. In this paper, we propose a top-view semantic completion network to produce accurate dense grid-wise predictions with real-time performance. Specifically, we propose an online distillation strategy, consisting of two parts: a student model using 2D range-view and top-view representations, and a teacher model using range-view, top-view, and voxel representations. To realize information transfer between different representations, we propose a cross-view association (CVA) module, by which the range-view features and 3D voxel features are converted into the ones in the top-view. The proposed method can avoid the difficulty of direct dense semantic segmentation in the top-view, with the point-wise sparse semantic segmentation module acting as a guide for the dense grid-wise semantic completion in a semantic-completion way. It can also alleviate the computational complexity by using only the voxel representation and 3D convolution in the teacher model. The experimental results on the SemanticKITTI dataset (46.4% mIoU) and nuScenes-LidarSeg dataset (47.3% mIoU) demonstrate the effectiveness of the proposed sparse guidance and online distillation strategies.

SOC Algorithm-Based Framework for Spectrum Occupancy Completion

This paper investigates the completion of continuous spectrum occupancy in space-frequency domain. Most of the existing studies adopt the traditional scheme, namely field strength completion followed by feature extraction. However, the existence of non-ideal sampling data challenges the recovery of spectrum occupancy array in accuracy. In this paper, we first develop a spectrum occupancy completion framework that implements feature extraction and spectrum occupancy completion sequentially, which reduces the impact of non-ideal data on completion. Then, to further improve the completion accuracy, a semantic-optional-based completion (SOC) algorithm is proposed, which divides the continuous spectrum occupancy data into discrete and multi-class spectrum semantic data according to the spectrum occupancy level. Specifically, the completion of spectrum occupancy is realized by specific semantic extraction, spectrum semantic completion and spectrum semantic recovery. Further, the real-world measured data are simulated to demonstrate the effectiveness of our proposed framework and SOC. The experimental results show that, compared with the low-rank completion algorithm and spatial interpolation algorithm, the proposed SOC has higher completion accuracy at more than 93% frequency points.

Semantic Completion and Filtration for Image–Text Retrieval

Image–text retrieval is a vital task in computer vision and has received growing attention, since it connects cross-modality data. It comes with the critical challenges of learning unified representations and eliminating the large gap between visual and textual domains. Over the past few decades, although many works have made significant progress in image–text retrieval, they are still confronted with the challenge of incomplete text descriptions of images, i.e., how to fully learn the correlations between relevant region–word pairs with semantic diversity. In this article, we propose a novel semantic completion and filtration (SCAF) method to alleviate the above issue. Specifically, the text semantic completion module is presented to generate a complete semantic description of an image using multi-view text descriptions, guiding the model to explore the correlations of relevant region–word pairs fully. Meanwhile, the adaptive structural semantic matching module is presented to filter irrelevant region–word pairs by considering the relevance score of each region–word pair, which facilitates the model to focus on learning the relevance of matching pairs. Extensive experiments show that our SCAF outperforms the existing methods on Flickr30K and MSCOCO datasets, which demonstrates the superiority of our proposed method.

SIGMA++: Improved Semantic-Complete Graph Matching for Domain Adaptive Object Detection.

Domain Adaptive Object Detection (DAOD) generalizes the object detector from an annotated domain to a label-free novel one. Recent works estimate prototypes (class centers) and minimize the corresponding distances to adapt the cross-domain class conditional distribution. However, this prototype-based paradigm 1) fails to capture the class variance with agnostic structural dependencies, and 2) ignores the domain-mismatched classes with a sub-optimal adaptation. To address these two challenges, we propose an improved SemantIc-complete Graph MAtching framework, dubbed SIGMA++, for DAOD, completing mismatched semantics and reformulating adaptation with hypergraph matching. Specifically, we propose a Hypergraphical Semantic Completion (HSC) module to generate hallucination graph nodes in mismatched classes. HSC builds a cross-image hypergraph to model class conditional distribution with high-order dependencies and learns a graph-guided memory bank to generate missing semantics. After representing the source and target batch with hypergraphs, we reformulate domain adaptation with a hypergraph matching problem, i.e., discovering well-matched nodes with homogeneous semantics to reduce the domain gap, which is solved with a Bipartite Hypergraph Matching (BHM) module. Graph nodes are used to estimate semantic-aware affinity, while edges serve as high-order structural constraints in a structure-aware matching loss, achieving fine-grained adaptation with hypergraph matching. The applicability of various object detectors verifies the generalization of SIGMA++, and extensive experiments on nine benchmarks show its state-of-the-art performance on both AP 50 and adaptation gains.

A Model of Semantic Completion in Generative Episodic Memory

Many studies have suggested that episodic memory is a generative process, but most computational models adopt a storage view. In this article, we present a model of the generative aspects of episodic memory. It is based on the central hypothesis that the hippocampus stores and retrieves selected aspects of an episode as a memory trace, which is necessarily incomplete. At recall, the neocortex reasonably fills in the missing parts based on general semantic information in a process we call semantic completion. The model combines two neural network architectures known from machine learning, the vector-quantized variational autoencoder (VQ-VAE) and the pixel convolutional neural network (PixelCNN). As episodes, we use images of digits and fashion items (MNIST) augmented by different backgrounds representing context. The model is able to complete missing parts of a memory trace in a semantically plausible way up to the point where it can generate plausible images from scratch, and it generalizes well to images not trained on. Compression as well as semantic completion contribute to a strong reduction in memory requirements and robustness to noise. Finally, we also model an episodic memory experiment and can reproduce that semantically congruent contexts are always recalled better than incongruent ones, high attention levels improve memory accuracy in both cases, and contexts that are not remembered correctly are more often remembered semantically congruently than completely wrong. This model contributes to a deeper understanding of the interplay between episodic memory and semantic information in the generative process of recalling the past.

Improved Semantic Inpainting Architecture Augmented with a Facial Landmark Detector

This paper presents an augmented method for image completion, particularly for images of human faces by leveraging on deep learning based inpainting techniques. Face completion generally tend to be a daunting task because of the relatively low uniformity of a face attributed to structures like eyes, nose, etc. Here, understanding the top level context is paramount for proper semantic completion. The method presented improves upon existing inpainting techniques that reduce context difference by locating the closest encoding of the damaged image in the latent space of a pre-trained deep generator. However, these existing methods fail to consider key facial structures (eyes, nose, jawline, etc.,) and their respective location to each other. This paper mitigates this by introducing a face landmark detector and a corresponding landmark loss. This landmark loss is added to the construction loss between the damaged and generated image and the adversarial loss of the generative model. The model was trained with the celeb A dataset, tools like pyamg, pillow and the OpenCV library was used for image manipulation and facial landmark detection. There are three main weighted parameters that balance the effect of the three loss functions in this paper, namely context loss, landmark loss and prior loss. Experimental results demonstrate that the added landmark loss attributes to better understanding of top-level context and hence the model can generate more visually appealing in painted images than the existing model.The model obtained average Structural Similarity Index (SSIM) and Peak Signal-to-Noise Ratio (PNSR) scores of 0.851 and 33.448 for different orientations of the face and 0.896 and 31.473, respectively, for various types masks.

GMatch: Knowledge base question answering via semantic matching

Effectiveness is essential for knowledge base question answering (KBQA) to determine whether the query can return the correct answers. Existing works for KBQA mainly focus on converting input questions into corresponding logic formats, such as SPARQL queries. However, since these works are largely decoupled from the knowledge base, the converted query may be ineffective. In this paper, we propose a novel semantic matching-based approach to model the query intention of the input question by extracting the subgraph of the knowledge base. The generation of the SPARQL query is reduced to semantic matching in the knowledge base to solve the ineffectiveness of the query. Firstly, a semantic query graph is proposed to model the reliable query intention of the input question. The SPARQL query graph could be extracted by matching the semantic query graph in the knowledge base. Secondly, an embedding-based method is developed to represent different forms of questions and queries in a common space. It is easy to detect semantic loss between the question and the converted query with the common representation. Finally, a data-driven semantic completion technique is presented to reduce the semantic loss by expanding the incomplete SPARQL query in the knowledge base. The experiments evaluated on benchmark datasets show that the proposed approach significantly outperforms state-of-the-art methods in efficiency and effectiveness. The code is available at https://github.com/offerhub-ht/gMatch.

Point Cloud Semantic Scene Completion from RGB-D Images

In this paper, we devise a novel semantic completion network, called point cloud semantic scene completion network (PCSSC-Net), for indoor scenes solely based on point clouds. Existing point cloud completion networks still suffer from their inability of fully recovering complex structures and contents from global geometric descriptions neglecting semantic hints. To extract and infer comprehensive information from partial input, we design a patch-based contextual encoder to hierarchically learn point-level, patch-level, and scene-level geometric and contextual semantic information with a divide-and-conquer strategy. Consider that the scene semantics afford a high-level clue of constituting geometry for an indoor scene environment, we articulate a semantics-guided completion decoder where semantics could help cluster isolated points in the latent space and infer complicated scene geometry. Given the fact that real-world scans tend to be incomplete as ground truth, we choose to synthesize scene dataset with RGB-D images and annotate complete point clouds as ground truth for the supervised training purpose. Extensive experiments validate that our new method achieves the state-of-the-art performance, in contrast with the current methods applied to our dataset.

Contextual Feature Constrained Semantic Face Completion With Paired Discriminator

Image semantic completion is to employ remaining image information to restore the damaged or missing areas. Face completion task is usually more challenging than other image inpainting problems as it requires stronger semantic consistency. We proposed a contextual feature constrained DCGAN with paired discriminator to inpaint damaged face images, which is capable of overcoming the DCGAN's shortages of insufficient feature learning capability and unstable training process. Our network is composed of an encoder-decoder generator and a local and global (paired) adversarial discriminator. Generator is used to produce missing parts, and the paired discriminators evaluate the authenticity of the local generated parts and the consistency of the global completed image respectively. In addition, the proposed network generates the image by optimizing the generator with three types of loss functions, i.e., image reconstruction loss, paired feature matching losses and paired feature reconstruction losses. The experiments on celebA and Stanford Cars dataset show that our model could generate reasonable repair results and that on some of the evaluation indicators were higher than baseline on most test datasets.

Weakly-Supervised Video Moment Retrieval via Semantic Completion Network

Video moment retrieval is to search the moment that is most relevant to the given natural language query. Existing methods are mostly trained in a fully-supervised setting, which requires the full annotations of temporal boundary for each query. However, manually labeling the annotations is actually time-consuming and expensive. In this paper, we propose a novel weakly-supervised moment retrieval framework requiring only coarse video-level annotations for training. Specifically, we devise a proposal generation module that aggregates the context information to generate and score all candidate proposals in one single pass. We then devise an algorithm that considers both exploitation and exploration to select top-K proposals. Next, we build a semantic completion module to measure the semantic similarity between the selected proposals and query, compute reward and provide feedbacks to the proposal generation module for scoring refinement. Experiments on the ActivityCaptions and Charades-STA demonstrate the effectiveness of our proposed method.

Attention-Based Multi-Modal Fusion Network for Semantic Scene Completion

This paper presents an end-to-end 3D convolutional network named attention-based multi-modal fusion network (AMFNet) for the semantic scene completion (SSC) task of inferring the occupancy and semantic labels of a volumetric 3D scene from single-view RGB-D images. Compared with previous methods which use only the semantic features extracted from RGB-D images, the proposed AMFNet learns to perform effective 3D scene completion and semantic segmentation simultaneously via leveraging the experience of inferring 2D semantic segmentation from RGB-D images as well as the reliable depth cues in spatial dimension. It is achieved by employing a multi-modal fusion architecture boosted from 2D semantic segmentation and a 3D semantic completion network empowered by residual attention blocks. We validate our method on both the synthetic SUNCG-RGBD dataset and the real NYUv2 dataset and the results show that our method respectively achieves the gains of 2.5% and 2.6% on the synthetic SUNCG-RGBD dataset and the real NYUv2 dataset against the state-of-the-art method.

RSDCN: A Road Semantic Guided Sparse Depth Completion Network

Laser radar (Lidar) plays an indispensable role in lots of security critical applications such as autonomous driving. However, the high sparsity and non-uniformity nature of the raw laser data brings large difficulties to reliable 3D scene understanding. Traditional depth completion methods suffer from the highly ill-conditioned nature of the problem. A novel end-to-end road semantic guided depth completion neural network with a special designed Asymmetric Multiscale Convolution (AMC) structure is proposed in this paper. The whole network is composed of two parts: semantic part and depth completion part. The semantic part is constructed by an image-Lidar joint segmentation sub-network which produces semantic masks (ground or object) to the following network. The depth completion part is composed of a series of AMC convolution structure. By combining the semantic masks and treating the ground and non-ground objects separately, the proposed AMC structure can well fit the depth distribution pattern implied in road scene. The experiments carried on both synthesized and real datasets demonstrate that our method can effectively improve the accuracy of depth completion results.

Learning pattern of hurricane damage levels using semantic web resources

This paper proposes an approach for hurricane damage level prediction using semantic web resources and matrix completion algorithms. Based on the statistical unit node set framework, streaming data from five hurricanes and damage levels from 48 counties in the USA were collected from the SRBench dataset and other web resources, and then trans-coded into matrices. At a time t, the pattern of possible highest damage levels at 6 hours into the future was estimated using a multivariate regression procedure based on singular value decomposition. We also applied soft-impute algorithm and k-nearest neighbours concept to improve the statistical unit node set framework in this research domain. Results showed that the model could deal with inaccurate, inconsistent and incomplete streaming data that were highly sparse, to learn future damage patterns and perform forecasting in near real-time. It was able to estimate the damage levels in several scenarios even if two-thirds of the relevant weather information was unavailable. The contributions of this work will be able to promote the applicability of the semantic web in the context of climate change.

Generative Steganography by Sampling

In this paper, a novel data-driven information hiding scheme called “generative steganography by sampling” (GSS) is proposed. Unlike in traditional modification-based steganography, in our method the stego image is directly sampled by a powerful generator: no explicit cover is used. Both parties share a secret key used for message embedding and extraction. The Jensen-Shannon divergence is introduced as a new criterion for evaluating the security of generative steganography. Based on these principles, we propose a simple practical generative steganography method that uses semantic image inpainting. The message is written in advance to an uncorrupted region that needs to be retained in the corrupted image. Then, the corrupted image with the secret message is fed into a Generator trained by a generative adversarial network (GAN) for semantic completion. Message loss and prior loss terms are proposed for penalizing message extraction error and unrealistic stego image. In our design, we first train a generator whose training target is the generation of new data samples from the same distribution as that of existing training data. Next, for the trained generator, backpropagation to the message and prior loss are introduced to optimize the coding of the input noise data for the generator. The presented experiments demonstrate the potential of the proposed framework based on both qualitative and quantitative evaluations of the generated stego images.

System description: Isabelle/jEdit in 2014

This is an updated system description for Isabelle/jEdit, according to the official release Isabelle2014 (August 2014). The following new PIDE concepts are explained: asynchronous print functions and document overlays, syntactic and semantic completion, editor navigation, management of auxiliary files within the document-model.

Mixed- versus strong right-handedness is associated with biases towards "remember" versus "know" judgements in recognition memory: Role of interhemispheric interaction

Growing evidence supports the hypothesis that episodic versus semantic memories rely primarily on interhemispheric versus intrahemispheric processing, respectively. For example, a recent study found that individuals with presumed greater interhemispheric interaction were superior in episodic recall but inferior at semantic word fragment completion; however, tests of recognition memory yielded no group differences. Interestingly, recognition memory can be based on either explicitly remembering a stimulus or implicitly knowing that a stimulus had been presented. The current experiments administered recognition memory tests to strongly versus mixed handed participants who judged for each recognised item whether their response was based on remembering (episodic memory) or knowing (semantic memory) (Tulving, 1983). Results indicate that strong versus mixed handers are biased towards basing recognition responses on judgements of knowing versus remembering, respectively. As strong versus mixed handedness is associated with greater versus lesser interhemispheric processing, the results support the original hypothesis.

Adaptive language and semantic models

In a spoken dialogue system, semantic interpretations are based on relations with words and phrases. Automatic learning of these relations is discussed and a solution based on semantic classification trees is presented. Interpretations can be complete if they provide the dialogue component with all the required information needed to produce an answer to the user. They can also be incomplete or there may be more than one complete or incomplete interpretations in competition, due to inherent ambiguities of the spoken message or to imprecision of the recognizer. Solutions for semantic completion and disambiguation will make dialogue systems more powerful and flexible. Preliminary results and future perspectives are discussed. Conceptual and language models (LM) can be dynamically adapted taking the dialogue history into account. Histories can be clustered into ‘‘dialogue situations’’ by considering the predicates used in the logic formula for generating an output message and selecting the one that is more likely to condition the answer. A dynamic probability can be expressed as combination of LM probabilities in various situations. Each probability is weighted according to the dialogue situation. Better LMs conceived with these principles will make future dialogue systems more robust.

Non-verbal rigidity, creativity, and problem solving.

Rigidity vs flexibility has often been mentioned in discussions of creativity and problem solving. The present study investigated the relation of a non-verbal test of rigidity (Breskin Rigidity Test) to tests of semantic redefinition, associational fluency, inductive reasoning, and drawing completion. The performance of flexible Ss was significantly better than the performance of rigid Ss on all tests but the associational fluency test.