Semantic Object Research Articles

SEMANTIC KNOWLEDGE EMBEDDING DEEP LEARNING NETWORK FOR LAND COVER CLASSIFICATION

Abstract. Land cover classification is essential basic information and key parameters for environmental change research, geographical and national monitoring, and sustainable development planning. Deep learning can automatically and multi-level extract the features of complex features, which has been proven to be an effective method for information extraction. However, one of the major challenges of deep learning is its poor interpret-ability, which makes it difficult to understand and explain the reasoning behind its classification results. This paper proposes a deep cross-modal coupling model (CMCM) for integrating semantic features and visual features. The representation of knowledge map is indicatively introduced into remote sensing image classification. Compared to previous studies, the proposed method provides accurate descriptions of the complex semantic objects within a complex land cover environment. The results showed that the integration of semantic knowledge improved the accuracy and interpret-ability of land cover classification.

VEHICLE OCCLUSION REMOVAL FROM SINGLE AERIAL IMAGES USING GENERATIVE ADVERSARIAL NETWORKS

Abstract. Removing occluding objects such as vehicles from drivable areas allows precise extraction of road boundaries and related semantic objects such as lane-markings, which is crucial for several applications such as generating high-definition maps for autonomous driving. Conventionally, multiple images of the same area taken at different times or from various perspectives are used to remove occlusions and to reconstruct the occluded areas. Nevertheless, these approaches require large amounts of data, which are not always available. Furthermore, they do not work for static occlusions caused by, among others, parked vehicles. In this paper, we address occlusion removal based on single aerial images using generative adversarial networks (GANs), which are able to deal with the mentioned challenges. To this end, we adapt several state-of-the-art GAN-based image inpainting algorithms to reconstruct the missing information. Results indicate that the StructureFlow algorithm outperforms the competitors and the restorations obtained are robust, with high visual fidelity in real-world applications. Furthermore, due to the lack of annotated aerial vehicle removal datasets, we generate a new dataset for training and validating the algorithms, the Aerial Vehicle Occlusion Removal (AVOR) dataset. To the best of our knowledge, our work is the first to address vehicle removal using deep learning algorithms to enhance maps.

Visual object detection using audio data

Nowadays Internet of Things (IoT) and Machine Learning (ML) are growing fields. One application of these two fields is object detection, which detects semantic objects using digital images and videos of classes like humans, vehicles, buildings, etc. Visual object detection systems are very effective and accurate due to the appearance information obtained from the cameras. But they face the problem of a limited Field of View. This paper aims to tackle this issue by using audio data to localize the object. A microphone is used to estimate the angular position of the object emitting the sound. Objects currently in the Field of View of a camera are detected and tracked using optical flow, but when they go out of the Field of View, the sound emitted by the object is used by the microphone to calculate the object‘s angular position. Once the angle is calculated, the camera is rotated in that direction. This thus ensures that the object can be located even if it goes out of frame. Once the object is located through IoT devices, we use ML to identify the person‘s face.

Low-Resolution Steel Surface Defects Classification Network Based on Autocorrelation Semantic Enhancement

Aiming at the problems of low-resolution steel surface defects imaging, such as defect type confusion, feature blurring, and low classification accuracy, this paper proposes an autocorrelation semantic enhancement network (ASENet) for the classification of steel surface defects. It mainly consists of a backbone network and an autocorrelation semantic enhancement module (ASE), in which the autocorrelation semantic enhancement module consists of three main learnable modules: the CS attention module, the autocorrelation computation module, and the contextual feature awareness module. Specifically, we first use the backbone network to extract the basic features of the image and then use the designed CS attention module to enhance the basic features. In addition, to capture different aspects of semantic objects, we use the autocorrelation module to compute the correlation between neighborhoods and contextualize the basic and augmented features to enhance the recognizability of the features. Experimental results show that our method produces significant results, and the classification accuracy reaches 96.24% on the NEU-CLS-64 dataset. Compared with ViT-B/16, Swin_t, ResNet50, Mobilenet_v3_small, Densenet121, Efficientnet_b2, and baseline, the accuracy is 9.43%, 5.15%, 4.87%, 3.34%, 3.28%, 3.01%, and 2.72% higher, respectively.

Recurrent connectivity supports higher-level visual and semantic object representations in the brain

Visual object recognition has been traditionally conceptualised as a predominantly feedforward process through the ventral visual pathway. While feedforward artificial neural networks (ANNs) can achieve human-level classification on some image-labelling tasks, it’s unclear whether computational models of vision alone can accurately capture the evolving spatiotemporal neural dynamics. Here, we probe these dynamics using a combination of representational similarity and connectivity analyses of fMRI and MEG data recorded during the recognition of familiar, unambiguous objects. Modelling the visual and semantic properties of our stimuli using an artificial neural network as well as a semantic feature model, we find that unique aspects of the neural architecture and connectivity dynamics relate to visual and semantic object properties. Critically, we show that recurrent processing between the anterior and posterior ventral temporal cortex relates to higher-level visual properties prior to semantic object properties, in addition to semantic-related feedback from the frontal lobe to the ventral temporal lobe between 250 and 500 ms after stimulus onset. These results demonstrate the distinct contributions made by semantic object properties in explaining neural activity and connectivity, highlighting it as a core part of object recognition not fully accounted for by current biologically inspired neural networks.

Semantic object processing is modulated by prior scene context

ABSTRACT Objects that are congruent with a scene are recognised more efficiently than objects that are incongruent. Further, semantic integration of incongruent objects elicits a stronger N300/N400 EEG component. Yet, the time course and mechanisms of how contextual information supports access to semantic object information is unclear. We used computational modelling and EEG to test how context influences semantic object processing. Using representational similarity analysis, we established that EEG patterns dissociated between objects in congruent or incongruent scenes from around 300 ms. By modelling the semantic processing of objects using independently normed properties, we confirm that the onset of semantic processing of both congruent and incongruent objects is similar (∼150 ms). Critically, after ∼275 ms, we discover a difference in the duration of semantic integration, lasting longer for incongruent compared to congruent objects. These results constrain our understanding of how contextual information supports access to semantic object information.

CDS-Net: Cooperative dual-stream network for image manipulation detection

To accurately locate manipulated regions, many existing approaches employ a dual-stream framework to extract a wide range of manipulation clues, including local noise, edge artifacts, and global inconsistency. However, these approaches treat each stream in isolation and fail to consider the complementary and mutual guidance ability between the streams. Moreover, we notice the use of vanilla vision transformers in previous approaches can result in disruptions of object semantics, causing incomplete predictions. To address these challenges, we introduce the cooperative dual-stream network (CDS-Net) comprising an RGB Stream and a Noise Stream. In the Noise Stream, we propose a K-means Transformer (KT) that encourages both inter-patch and intra-patch information transmission to mitigate the semantic fragmentation phenomenon caused by patch partitioning. Additionally, we introduce a novel Feature Interaction Block (FIB) that explicitly encourages cross-stream collaboration at each encoding stage. Comprehensive experiments on publicly available datasets demonstrate the effectiveness and robustness of CDS-Net.

Image emotion multi-label classification based on multi-graph learning

Images contain rich information and can induce various emotions in the audience. Image emotion classification aims to identify the emotion categories that images can evoke. It is widely used in mental health assessment, human–computer interaction, etc. There are two main problems in existing image emotion classification methods: (1) Most of them only focus on a single emotion label; (2) The global structural relationship among semantic objects in the image is ignored. Therefore, this paper proposes an Image Emotion classification method based on Multi-Graph Multi-Label learning (IE-MGML). In contrast to the existing approaches, the image is transformed into a graph-based representation by extracting the emotional features of semantic objects and calculating the similarity between the features. The local (semantic objects) features and global structure (relationship among semantic objects) features of the image are fused by the relationship between nodes. Furthermore, the graph representation of an image from the perspective of multiple emotional features is pooled and modeled as a graph bag containing multiple graphs (i.e., multi-graph). In multi-graph learning, the graph kernel directly evaluates a graph-label dependency score to avoid the loss of structural information caused by graph-instance degradation. The bag(image)-label dependency score is obtained by aggregating the graph-label dependency score from different perspectives through the aggregation function. The problem of error accumulation in the learning process is handled by proposing a threshold-based ranking loss objective function. Moreover, the non-convex optimization problem is addressed using a subgradient descent algorithm to deal with the required high-dimensional space computation. Experimental results on three general image emotion datasets show that the proposed method outperforms the state-of-the-art methods.

Accurate Perception and Association of Objects for Humanoid Robots Under Dynamic Visual SLAM

Humanoid robots still have a low-level perception of their surroundings, which is a formidable obstacle to performing complex tasks. To improve their perceptual capabilities, high-level semantic information can be incorporated into their perceptual framework so that they can gain the ability to infer and understand their environment accurately. Among the various techniques available, semantic Simultaneous Localization and Mapping (SLAM) becomes a promising avenue for achieving this perception enhancement. While semantic SLAM can enhance a robot’s ability to perform tasks, the accurate perception, and association of semantic objects in complex and dynamic environments remain challenging. Hence, the need arises for a solution that swiftly and precisely associates object measurements with landmarks, considering their motion properties, and promptly rectifies erroneous associations in real time. To this end, we propose a semantic perception approach designed explicitly for dynamic environments, adept at distinguishing between dynamic and static objects. Furthermore, we propose two association strategies: dynamic object association based on semantic map points and static object association based on object pose information. And as the number of object measurements associated with the landmarks increases, we perform the association validation algorithm to verify the association for the landmark to improve the accuracy of the association. The proposed method is extensively evaluated on both simulated indoor sequences obtained from humanoid robot viewpoints and the KITTI dataset. Experimental results show that our approach significantly improves the robustness and accuracy of object association and trajectory estimation.

Inferring Attention Shifts for Salient Instance Ranking

AbstractThe human visual system has limited capacity in simultaneously processing multiple visual inputs. Consequently, humans rely on shifting their attention from one location to another. When viewing an image of complex scenes, psychology studies and behavioural observations show that humans prioritise and sequentially shift attention among multiple visual stimuli. In this paper, we propose to predict the saliency rank of multiple objects by inferring human attention shift. We first construct a new large-scale salient object ranking dataset, with the saliency rank of objects defined by the order that an observer attends to these objects via attention shift. We then propose a new deep learning-based model to leverage both bottom-up and top-down attention mechanisms for saliency rank prediction. Our model includes three novel modules: Spatial Mask Module (SMM), Selective Attention Module (SAM) and Salient Instance Edge Module (SIEM). SMM integrates bottom-up and semantic object properties to enhance contextual object features, from which SAM learns the dependencies between object features and image features for saliency reasoning. SIEM is designed to improve segmentation of salient objects, which helps further improve their rank predictions. Experimental results show that our proposed network achieves state-of-the-art performances on the salient object ranking task across multiple datasets. Code and data are available at https://github.com/SirisAvishek/Attention_Shift_Ranks.

Spatiotemporal characteristics of the neural representation of event concepts

Events are a fundamentally important part of our understanding of the world. How lexical concepts denoting events are represented in the brain remains controversial. We conducted two experiments using event and object nouns matched on a range of psycholinguistic variables, including concreteness, to examine spatial and temporal characteristics of event concepts. Both experiments used magnitude and valence tasks on event and object nouns. The fMRI experiment revealed a distributed set of regions for events, including the angular gyrus, anterior temporal lobe, and posterior cingulate across tasks. In the EEG experiment, events and objects differed in amplitude within the 300–500 ms window. Together these results shed light into the spatiotemporal characteristics of event concept representation and show that event concepts are represented in the putative hubs of the semantic system. While these hubs are typically associated with object semantics, they also represent events, and have a likely role in temporal integration.

Types and Type Theories in Natural Language Analysis

This article reviews the set of possible paths from a semantics based on Simple Type Theories (STTs) toward one based on Rich Type Theories (RTTs) and the motivations behind the move from one to the other. The main elements of this review are threefold. First, it provides a systematic overview of different STTs, including options for what to include as members of the set of basic types, and whether to assume type constructors additional to the one for constructing functional types. Second, this review discusses the main differences between STTs and RTTs, namely, that in RTTs but not in STTs, types are part of the object language. That is, one can refer to and reason with and about types. In turn, this makes available an alternative account of propositions to the one assumed in semantics in the Frege–Church–Montague tradition: Instead of being characterized as sets of possible worlds, propositions can be treated themselves as types, that is, as structured semantic objects. Third and finally, this review provides an outline of some of the main applications of RTTs, including hyperintensionality, quantification, anaphora, polysemy, and modification.

Pre-Training Across Different Cities for Next POI Recommendation

The Point-of-Interest (POI) transition behaviors could hold absolute sparsity and relative sparsity very differently for different cities. Hence, it is intuitive to transfer knowledge across cities to alleviate those data sparsity and imbalance problems for next POI recommendation. Recently, pre-training over a large-scale dataset has achieved great success in many relevant fields, like computer vision and natural language processing. By devising various self-supervised objectives, pre-training models can produce more robust representations for downstream tasks. However, it is not trivial to directly adopt such existing pre-training techniques for next POI recommendation, due to the lacking of common semantic objects (users or items) across different cities . Thus in this paper, we tackle such a new research problem of pre-training across different cities for next POI recommendation. Specifically, to overcome the key challenge that different cities do not share any common object, we propose a novel pre-training model named CATUS , by transferring the cat egory-level u niversal tran s ition knowledge over different cities. Firstly, we build two self-supervised objectives in CATUS : next category prediction and next POI prediction , to obtain the universal transition-knowledge across different cities and POIs. Then, we design a category-transition oriented sampler on the data level and an implicit and explicit transfer strategy on the encoder level to enhance this transfer process. At the fine-tuning stage, we propose a distance oriented sampler to better align the POI representations into the local context of each city. Extensive experiments on two large datasets consisting of four cities demonstrate the superiority of our proposed CATUS over the state-of-the-art alternatives. The code and datasets are available at https://github.com/NLPWM-WHU/CATUS.

Deep Surveillance System

This study has been undertaken to investigate and implement multiple detection systems into a single surveillance system and check whether the input videos may comprise and capture a variety of realistic anomalies or not. In this paper, we propose to learn various anomalies by exploiting both normal and anomalous videos and implemented it to new model. Real time object detection is a vast, vibrant and sophisticated area of computer vision aimed towards object identification and recognition. Object detection detects the semantic objects of a class objects using Open source Computer Vision, which is a library of programming functions mainly trained towards real time computer vision in digital images and videos. The main aim behind this real time object detection is to help the peoples to overcome their difficulty. Real time object detection finds its uses in the areas like tracking objects, video surveillance, pedestrian detection, people counting, self-driving cars, face detection, tracking in sports and many more. This is achieved using Convolution, Probabilistic Neural Networks, etc. which are a representative tool of Deep learning. This project acts as an aiding tool for peoples who wants to take care of everything inside, outside, and around their house just for their full security expectations. Surveillance is a must for small houses to large-scale industries as they fulfil our safety aspects because theft and burglary have always been a problem. By combining this Surveillance idea to IoT and some Machine Learning stuff this will be a major product. The proposed project is a single autonomous surveillance system, based on analysis and detection technology. The proposed system is capable of monitoring all actions at once and alerts the concerned officials immediately and precisely.

MISL: Multi-grained image-text semantic learning for text-guided image inpainting

Text-guided image inpainting aims to generate corrupted image patches and obtain a plausible image based on textual descriptions, considering the relationship between textual and visual semantics. Existing works focus on predicting missing patches from the residual pixels of corrupted images, ignoring the visual semantics of the objects of interest in the images corresponding to the textual descriptions. In this paper, we propose a text-guided image inpainting method with multi-grained image-text semantic learning (MISL), consisting of global-local generators and discriminators. More specifically, we devise hierarchical learning (HL) with global-coarse-grained, object-fine-grained, and global-fine-grained learning stages in the global-local generators to refine the corrupted images from the global to local. In particular, the object-fine-grained learning stage focuses on the visual semantics of objects of interest in corrupted images by using an encoder-decoder network with self-attention blocks. Not only that, we design a mask reconstruction (MR) module to further act on the restoration of the objects of interest corresponding to the given textual descriptions. To inject the textual semantics into the global-local generators, we implement a multi-attention (MA) module that incorporates the word-level and sentence-level textual features to generate three different-grained images. For training, we exploit a global discriminator and a flexible discriminator to penalize the whole image and the corrupted region, respectively. Extensive experiments conducted on four datasets show the outperformance of the proposed MISL.

Global enhancement network underwater archaeology scene parsing method

AbstractUnderwater archaeology is of great significance for historical and cultural transmission and preservation of underwater heritage, but it is also a challenging task. Underwater heritage is located in an environment with high sediment content, objects are mostly buried, and the water is turbid, resulting in some of the features of objects missing or blurred, making it difficult to accurately identify and understand the semantics of various objects in the scene. To tackle these issues, this paper proposes a global enhancement network (GENet) underwater scene parsing method. We introduce adaptive dilated convolution by adding an extra regression layer, which can automatically deduce adaptive dilated coefficients according to the different scene objects. In addition, considering the easy confusion in the process of fuzzy feature classification, an enhancement classification network is proposed to increase the difference between various types of probabilities by reducing the loss function. We verified the validity of the proposed model by conducting numerous experiments on the Underwater Shipwreck Scenes (USS) dataset. We achieve state-of-the-art performance compared to the current state-of-the-art algorithm under three different conditions: conventional, relic semi-buried, and turbidified water quality. The experimental results show that the proposed algorithm performs best in different situations. To verify the generalizability of the proposed algorithm, we conducted comparative experiments on the current publicly available Cityscapes, ADE20K, and the underwater dataset SUIM. The experimental results show that this paper achieves good performance on the public dataset, indicating that the proposed algorithm is generalizable.

S3R: Shape and Semantics-Based Selective Regularization for Explainable Continual Segmentation Across Multiple Sites.

In clinical practice, it is desirable for medical image segmentation models to be able to continually learn on a sequential data stream from multiple sites, rather than a consolidated dataset, due to storage cost and privacy restrictions. However, when learning on a new site, existing methods struggle with a weak memorizability for previous sites with complex shape and semantic information, and a poor explainability for the memory consolidation process. In this work, we propose a novel Shape and Semantics-based Selective Regularization ( [Formula: see text]) method for explainable cross-site continual segmentation to maintain both shape and semantic knowledge of previously learned sites. Specifically, [Formula: see text] method adopts a selective regularization scheme to penalize changes of parameters with high Joint Shape and Semantics-based Importance (JSSI) weights, which are estimated based on the parameter sensitivity to shape properties and reliable semantics of the segmentation object. This helps to prevent the related shape and semantic knowledge from being forgotten. Moreover, we propose an Importance Activation Mapping (IAM) method for memory interpretation, which indicates the spatial support for important parameters to visualize the memorized content. We have extensively evaluated our method on prostate segmentation and optic cup and disc segmentation tasks. Our method outperforms other comparison methods in reducing model forgetting and increasing explainability. Our code is available at https://github.com/jingyzhang/S3R.

Probabilistic Approach for Road-Users Detection

Object detection in autonomous driving applications implies the detection and tracking of semantic objects that are commonly native to urban driving environments, as pedestrians and vehicles. One of the major challenges in state-of-the-art deep-learning based object detection are false positives which occur with overconfident scores. This is highly undesirable in autonomous driving and other critical robotic-perception domains because of safety concerns. This paper proposes an approach to alleviate the problem of overconfident predictions by introducing a novel probabilistic layer to deep object detection networks in testing. The suggested approach avoids the traditional Sigmoid or Softmax prediction layer which often produces overconfident predictions. It is demonstrated that the proposed technique reduces overconfidence in the false positives without degrading the performance on the true positives. The approach is validated on the 2D-KITTI objection detection through the YOLOV4 and SECOND (Lidar-based detector). The proposed approach enables interpretable probabilistic predictions without the requirement of re-training the network and therefore is very practical.

A Comparative Study of Object Detection using YOLO and SSD Algorithms

One of the most significant and difficult areas of computer vision is object detection, which is used in many aspects of daily life, including security monitoring, autonomous vehicle operation, and so on, to find instances of semantic objects belonging to a particular class. In this survey, we first look at the current approaches of typical detection models and discuss the benchmark datasets in order to comprehensively and deeply comprehend the major development status of the object detection pipeline. This project explains how convolutional neural network-based deep learning techniques are used for object detection. In this study, deep learning methods for cutting-edge object identification systems are evaluated. Any kind of visual medium is represented by a computer as a collection of numerical numbers. In order to inspect the contents of images, they need image processing techniques. In order to determine which is the quickest and most effective object detection algorithm, this research analyzes two popular algorithms: Single Shot Detection (SSD) and You Only Look Once (YOLO). The COCO (Common Object in Context) dataset is used in this comparative research to assess the effectiveness of the two algorithms and to analyze their strengths and weaknesses in terms of factors like accuracy, precision, and speed. Based on the findings analysis, it can be said that YOLO-v3 outperforms SSD in the same testing scenario. Keywords- Object Detection, Deep learning, CNN, COCO, YOLO, SSD

Semantic-Aware Contrastive Learning for Multi-Object Medical Image Segmentation.

Medical image segmentation, or computing voxel-wise semantic masks, is a fundamental yet challenging task in medical imaging domain. To increase the ability of encoder-decoder neural networks to perform this task across large clinical cohorts, contrastive learning provides an opportunity to stabilize model initialization and enhances downstream tasks performance without ground-truth voxel-wise labels. However, multiple target objects with different semantic meanings and contrast level may exist in a single image, which poses a problem for adapting traditional contrastive learning methods from prevalent "image-level classification" to "pixel-level segmentation". In this article, we propose a simple semantic-aware contrastive learning approach leveraging attention masks and image-wise labels to advance multi-object semantic segmentation. Briefly, we embed different semantic objects to different clusters rather than the traditional image-level embeddings. We evaluate our proposed method on a multi-organ medical image segmentation task with both in-house data and MICCAI Challenge 2015 BTCV datasets. Compared with current state-of-the-art training strategies, our proposed pipeline yields a substantial improvement of 5.53% and 6.09% on Dice score for both medical image segmentation cohorts respectively (p-value 0.01). The performance of the proposed method is further assessed on external medical image cohort via MICCAI Challenge FLARE 2021 dataset, and achieves a substantial improvement from Dice 0.922 to 0.933 (p-value 0.01).