Scene Semantic Information Research Articles

A three-dimensional feature-based fusion strategy for infrared and visible image fusion

Due to the lacking of attention to the scene’s essential characteristics, the existing fusion methods suffer from the deficiency of scene distortion. In addition, the lack of groundtruth can cause an inadequate representation of vital information. To this end, we propose a novel infrared and visible image fusion network based on three-dimensional feature fusion strategy (D3Fuse). In our method, we consider the scene semantic information in the source images and extract the commonality contents of the two images as the third-dimensional feature to extend the feature space for fusion tasks. Specifically, a commonality feature extraction module (CFEM) is designed to extract the scene commonality features. Subsequently, the scene commonality features are utilized together with modality features to construct the fusion image. Moreover, to ensure the independence and diversity of distinct features, we employ a contrastive learning strategy with multiscale PCA coding, which stretches the feature distance in an unsupervised manner, prompting the encoder to extract more discriminative information without incurring additional parameters and computational costs. Furthermore, a contrastive enhancement strategy is utilized to ensure adequate representation of modality information. The results of the qualitative and quantitative evaluations on the three datasets show that the proposed method has better visual performance and higher objective metrics with lower computational cost. The object detection experiments show that our results have superior performance on high-level semantic tasks.

3D Surface Reconstruction Based on Dynamic Graph Convolutional Occupancy Network

A 3D reconstruction method based on dynamic graph convolutional occupancy networks is proposed to address the issues of texture information loss, geometric information loss after voxelization, and lack of object completeness constraints in the process of 3D reconstruction using voxel representation in a block-wise manner. By constructing a dynamic graph structure for feature extraction, the method aims at restore 3D models with fewer holes and local details. In the feature extraction stage, local pooling is employed within each point cloud block to address the problem of nonsignificant texture feature loss. To tackle the issues of geometric constraint loss and insufficient scene semantic information caused by block-wise processing, a feature fusion method between adjacent blocks is proposed to learn richer scene semantic information and long-range dependencies between points. By learning features within and between blocks, each point retains as much geometric information as possible, mitigating the problem of geometric information loss due to voxelization. During the surface generation, interpolation is used to infer the occupancy value for each point, and the Marching Cubes algorithm is employed for three-dimensional surface reconstruction. Experimental validation on object-level (ShapeNet dataset) and scene-level (Synthetic Rooms dataset, MatterPort3D dataset for real-world scenes) datasets demonstrates the effectiveness and advancement of the proposed method.

Dynamic Speed and Separation Monitoring Based on Scene Semantic Information

Dynamic Speed and Separation Monitoring Based on Scene Semantic Information

A semantic-centered cloud control framework for autonomous unmanned system

Rich semantic information in natural language increases team efficiency in human collaboration, reduces dependence on high precision data information, and improves adaptability to dynamic environment. We propose a semantic centered cloud control framework for cooperative multi-unmanned ground vehicle (UGV) system. Firstly, semantic modeling of task and environment is implemented by ontology to build a unified conceptual architecture, and secondly, a scene semantic information extraction method combining deep learning and semantic web rule language (SWRL) rules is used to realize the scene understanding and task-level cloud task cooperation. Finally, simulation results show that the framework is a feasible way to enable autonomous unmanned systems to conduct cooperative tasks.

RelationRS: Relationship Representation Network for Object Detection in Aerial Images

Object detection is a basic and important task in the field of aerial image processing and has gained much attention in computer vision. However, previous aerial image object-detection approaches have insufficient use of scene semantic information between different regions of large-scale aerial images. In addition, complex background and scale changes make it difficult to improve detection accuracy. To address these issues, we propose a relationship representation network for object detection in aerial images (RelationRS): (1) Firstly, multi-scale features are fused and enhanced by a dual relationship module (DRM) with conditional convolution. The dual relationship module learns the potential relationship between features of different scales and learns the relationship between different scenes from different patches in a same iteration. In addition, the dual relationship module dynamically generates parameters to guide the fusion of multi-scale features. (2) Secondly, the bridging visual representations module (BVR) is introduced into the field of aerial images to improve the object detection effect in images with complex backgrounds. Experiments with a publicly available object detection dataset for aerial images demonstrate that the proposed RelationRS achieves a state-of-the-art detection performance.

FEATURE FUSION FOR CROSS-MODAL SCENE CLASSIFICATION OF REMOTE SENSING IMAGE

Abstract. Scene classification plays an important role in remote sensing field. Traditional approaches use high-resolution remote sensing images as data source to extract powerful features. Although these kind of methods are common, the model performance is severely affected by the image quality of the dataset, and the single modal (source) of images tend to cause the mission of some scene semantic information, which eventually degrade the classification accuracy. Nowadays, multi-modal remote sensing data become easy to obtain since the development of remote sensing technology. How to carry out scene classification of cross-modal data has become an interesting topic in the field. To solve the above problems, this paper proposes using feature fusion for cross-modal scene classification of remote sensing image, i.e., aerial and ground street view images, expecting to use the advantages of aerial images and ground street view data to complement each other. Our cross- modal model is based on Siamese Network. Specifically, we first train the cross-modal model by pairing different sources of data with aerial image and ground data. Then, the trained model is used to extract the deep features of the aerial and ground image pair, and the features of the two perspectives are fused to train a SVM classifier for scene classification. Our approach has been demonstrated using two public benchmark datasets, AiRound and CV-BrCT. The preliminary results show that the proposed method achieves state-of-the-art performance compared with the traditional methods, indicating that the information from ground data can contribute to aerial image classification.

OBJECT RECOGNITION FOR AUGMENTED REALITY APPLICATIONS

Developments in the field of neural networks, deep learning, and increases in computing systems’ capacity have allowed for a significant performance boost in scene semantic information extraction algorithms and their respective mechanisms. The work presented in this paper investigates the performance of various object classification- recognition frameworks and proposes a novel framework, which incorporates Super-Resolution as a preprocessing method, along with YOLO/Retina as the deep neural network component. The resulting scene analysis framework was fine-tuned and benchmarked using the COCO dataset, with the results being encouraging. The presented framework can potentially be utilized, not only in still image recognition scenarios but also in video processing.

Integrating Scene Semantic Knowledge into Image Captioning

Most existing image captioning methods use only the visual information of the image to guide the generation of captions, lack the guidance of effective scene semantic information, and the current visual attention mechanism cannot adjust the focus intensity on the image. In this article, we first propose an improved visual attention model. At each timestep, we calculated the focus intensity coefficient of the attention mechanism through the context information of the model, then automatically adjusted the focus intensity of the attention mechanism through the coefficient to extract more accurate visual information. In addition, we represented the scene semantic knowledge of the image through topic words related to the image scene, then added them to the language model. We used the attention mechanism to determine the visual information and scene semantic information that the model pays attention to at each timestep and combined them to enable the model to generate more accurate and scene-specific captions. Finally, we evaluated our model on Microsoft COCO (MSCOCO) and Flickr30k standard datasets. The experimental results show that our approach generates more accurate captions and outperforms many recent advanced models in various evaluation metrics.

Hierarchical Parsing Net: Semantic Scene Parsing From Global Scene to Objects

This paper proposes a novel Hierarchical Parsing Net (HPN) for semantic scene parsing. Unlike previous methods, which separately classify each object, HPN leverages global scene semantic information and the context among multiple objects to enhance scene parsing. On the one hand, HPN uses the global scene category to constrain the semantic consistency between the scene and each object. On the other hand, the context among all objects is also modeled to avoid incompatible object predictions. Specifically, HPN consists of four steps. In the first step, we extract scene and local appearance features. Based on these appearance features, the second step is to encode a contextual feature for each object, which models both the scene-object context (the context between the scene and each object) and the interobject context (the context among different objects). In the third step, we classify the global scene and then use the scene classification loss and a backpropagation algorithm to constrain the scene feature encoding. In the fourth step, a label map for scene parsing is generated from the local appearance and contextual features. Our model outperforms many state-of-the-art deep scene parsing networks on five scene parsing databases.

Visual Saliency Prediction Using a Mixture of Deep Neural Networks.

Visual saliency models have recently begun to incorporate deep learning to achieve predictive capacity much greater than previous unsupervised methods. However, most existing models predict saliency without explicit knowledge of global scene semantic information. We propose a model (MxSalNet) that incorporates global scene semantic information in addition to local information gathered by a convolutional neural network. Our model is formulated as a mixture of experts. Each expert network is trained to predict saliency for a set of closely related images. The final saliency map is computed as a weighted mixture of the expert networks' output, with weights determined by a separate gating network. This gating network is guided by global scene information to predict weights. The expert networks and the gating network are trained simultaneously in an end-toend manner. We show that our mixture formulation leads to improvement in performance over an otherwise identical nonmixture model that does not incorporate global scene information. Additionally, we show that our model achieves better performance than several other visual saliency models.

Use of Roadway Scene Semantic Information and Geometry-Preserving Landmark Pairs to Improve Visual Place Recognition in Changing Environments

Visual place recognition (VPR) in changing environments is an urgent challenge for long-term autonomous navigation. One recent ConvNet landmark-based approach exploits region landmarks coupled with ConvNet features to match images, and the approach has shown promising results under significant environmental and viewpoint changes. In this paper, we propose a robust ConvNet landmark-based system for VPR in changing outdoor roadway environments by extension of this approach from the following two aspects. First, our method utilizes more discriminative landmarks obtained by a novel refinement method called SemLandmarks , which leverages roadway scene semantic information to screen landmarks directly detected by an existing object proposal method. Second, our method improves the accuracy of image matching by introducing consistent spatial constraints based on the use of geometry-preserving landmark pairs. Experimental results demonstrate that our method significantly improves the state of the art in VPR in terms of recognition accuracy on three challenging benchmark data sets with various environmental and viewpoint changes.

Scene context influences without scene gist: Eye movements guided by spatial associations in visual search

Although the use of semantic information about the world seems ubiquitous in every task we perform, it is not clear whether we rely on a scene's semantic information to guide attention when searching for something in a specific scene context (e.g., keys in one's living room). To address this question, we compared contribution of a scene's semantic information (i.e., scene gist) versus learned spatial associations between objects and context. Using the flash-preview-moving-window paradigm Castelhano and Henderson (Journal of Experimental Psychology: Human Perception and Performance 33:753-763, 2007), participants searched for target objects that were placed in either consistent or inconsistent locations and were semantically consistent or inconsistent with the scene gist. The results showed that learned spatial associations were used to guide search even in inconsistent contexts, providing evidence that scene context can affect search performance without consistent scene gist information. We discuss the results in terms of hierarchical organization of top-down influences of scene context.