Holistic Scene Research Articles

SEMScene: Semantic-Consistency Enhanced Multi-Level Scene Graph Matching for Image-Text Retrieval

Image-text retrieval, a fundamental cross-modal task, performs similarity reasoning for images and texts. The primary challenge for image-text retrieval is cross-modal semantic heterogeneity, where the semantic features of visual and textual modalities are rich but distinct. Scene graph is an effective representation for images and texts as it explicitly models objects and their relations. Existing scene graph based methods have not fully taken the features regarding various granularities implicit in scene graph into consideration (e.g. triplets), the inadequate feature matching incurs the absence of non-trivial semantic information (e.g. inner relations among triplets). Therefore, we propose a S emantic-Consistency E nhanced M ulti-Level Scene Graph Matching (SEMScene) network, which exploits the semantic relevance between visual and textual scene graphs from fine-grained to coarse-grained. Firstly, under the scene graph representation, we perform feature matching including low-level node matching, mid-level semantic triplet matching, and high-level holistic scene graph matching. Secondly, to enhance the semantic-consistency for object-fused triplets carrying key correlation information, we propose a dual-step constraint mechanism in mid-level matching. Thirdly, to guide the model to learn the semantic-consistency of matched image-text pairs, we devise effective loss functions for each stage of the dual-step constraint. Comprehensive experiments on Flickr30K and MS-COCO datasets demonstrate that SEMScene achieves state-of-the-art performances with significant improvements.

Sequential selection and calibration of video frames for 3D outdoor scene reconstruction

Abstract3D scene understanding and reconstruction aims to obtain a concise scene representation from images and reconstruct the complete scene, including the scene layout, objects bounding boxes and shapes. Existing holistic scene understanding methods primarily recover scenes from single images, with a focus on indoor scenes. Due to the complexity of real‐world, the information provided by a single image is limited, resulting in issues such as object occlusion and omission. Furthermore, captured data from outdoor scenes exhibits characteristics of sparsity, strong temporal dependencies and a lack of annotations. Consequently, the task of understanding and reconstructing outdoor scenes is highly challenging. The authors propose a sparse multi‐view images‐based 3D scene reconstruction framework (SMSR). It divides the scene reconstruction task into three stages: initial prediction, refinement, and fusion stage. The first two stages extract 3D scene representations from each viewpoint, while the final stage involves selection, calibration and fusion of object positions and orientations across different viewpoints. SMSR effectively address the issue of object omission by utilizing small‐scale sequential scene information. Experimental results on the general outdoor scene dataset UrbanScene3D‐Art Sci and our proprietary dataset Software College Aerial Time‐series Images, demonstrate that SMSR achieves superior performance in the scene understanding and reconstruction.

Contextualized Relation Predictive Model for Self-Supervised Group Activity Representation Learning

Group activity analysis has attracted remarkable attention recently due to the widespread applications in security, entertainment and military. This paper targets at learning group activity representations with self-supervision, which differs from the majorities relying heavily on manually annotated labels. Moreover, existing Self-Supervised Learning (SSL) methods for videos are sub-optimal to generate such representations because of the complex context dynamics in group activities. In this paper, an end-to-end framework termed Contextualized Relation Predictive Model (Con-RPM) is proposed for self-supervised group activity representation learning with predictive coding. It involves the Serial-Parallel Transformer Encoder (SPTrans-Encoder) to model the context of spatial interactions and temporal variations, and the Hybrid Context Transformer Decoder (HConTrans-Decoder) to predict the future spatio-temporal relations guided by holistic scene context. Additionally, to improve the discriminability and consistency of prediction, we introduce a united loss integrating group-wise and person-wise contrastive losses in frame-level as well as the adversarial loss in global sequence-level. Consequently, our Con-RPM learns robust group representations via describing temporal evolutions of individual relationships and scene semantics explicitly. Extensive experimental results on downstream tasks indicate the effectiveness and generalization of our model in self-supervised learning, and present state-of-the-art performance on the Volleyball, Collective Activity, VolleyTactic, and Choi's New datasets.

Variable-length sequence model for attribute detection in the image

Holistic scene understanding is a challenging problem in computer vision. Most recent researches in this field were focusing on the object detection, the semantic segmentation and the relationship detection tasks. The attribute can provide meaningful information for the object instance, thus the object instance can be expressed more detail in the scene understanding. However, most researches in this field have been limited to several special conditions. Such as, several researches were just focusing on the attribute of special object class, because their solutions were aimed at a limited-scenarios, their methods are hardly to generalize in other scenarios. We also find that most of the research for multi-attribute detection task were only regarding each attribute as binary class and simply use the multi-binary-classifier method for the attribute detection. But these strategies above not consider the relation between each pair of the attributes, they will fall into trouble in the “imperfect” attribute dataset (which is labeled with the missing and incomplete annotations), and they will have low performance in the long-tail attribute class (which has lower rank of annotation and more missing labels). In this paper, we focus on the multi-attribute detection for a variant of object classes and take the relation between attributes into consideration. We propose a GRU-based model to detect a variable-length attribute sequence with a customized loss compute method to solve the “imperfect” attribute dataset problem. Furthermore, we perform ablative studies to prove the effectiveness of each part of our method. Finally, we compare our model with several existed multi-attribute detection methods on VG (Visual Genome) and CUB200 bird datasets to prove the superior performance of the proposed model.

Multi-view Consistent Generative Adversarial Networks for Compositional 3D-Aware Image Synthesis

This paper studies compositional 3D-aware image synthesis for both single-object and multi-object scenes. We observe that two challenges remain in this field: existing approaches (1) lack geometry constraints and thus compromise the multi-view consistency of the single object, and (2) can not scale to multi-object scenes with complex backgrounds. To address these challenges coherently, we propose multi-view consistent generative adversarial networks (MVCGAN) for compositional 3D-aware image synthesis. First, we build the geometry constraints on the single object by leveraging the underlying 3D information. Specifically, we enforce the photometric consistency between pairs of views, encouraging the model to learn the inherent 3D shape. Second, we adapt MVCGAN to multi-object scenarios. In particular, we formulate the multi-object scene generation as a “decompose and compose” process. During training, we adopt the top-down strategy to decompose training images into objects and backgrounds. When rendering, we deploy a reverse bottom-up manner by composing the generated objects and background into the holistic scene. Extensive experiments on both single-object and multi-object datasets show that the proposed method achieves competitive performance for 3D-aware image synthesis.

Panoramic Panoptic Segmentation: Insights Into Surrounding Parsing for Mobile Agents via Unsupervised Contrastive Learning

In this work, we introduce panoramic panoptic segmentation, as the most holistic scene understanding, both in terms of Field of View (FoV) and image-level understanding for standard camera-based input. A complete surrounding understanding provides a maximum of information to a mobile agent. This is essential information for any intelligent vehicle to make informed decisions in a safety-critical dynamic environment such as real-world traffic. In order to overcome the lack of annotated panoramic images, we propose a framework which allows model training on standard pinhole images and transfers the learned features to the panoramic domain in a cost-minimizing way. The domain shift from pinhole to panoramic images is non-trivial as large objects and surfaces are heavily distorted close to the image border regions and look different across the two domains. Using our proposed method with dense contrastive learning, we manage to achieve significant improvements over a non-adapted approach. Depending on the efficient panoptic segmentation architecture, we can improve 3.5–6.5% measured in Panoptic Quality (PQ) over non-adapted models on our established Wild Panoramic Panoptic Segmentation (WildPPS) dataset. Furthermore, our efficient framework does not need access to the images of the target domain, making it a feasible domain generalization approach suitable for a limited hardware setting. As additional contributions, we publish WildPPS: The first panoramic panoptic image dataset to foster progress in surrounding perception and explore a novel training procedure combining supervised and contrastive training.

Im2Graph: A Weakly Supervised Approach for Generating Holistic Scene Graphs from Regional Dependencies

Conceptual representations of images involving descriptions of entities and their relations are often represented using scene graphs. Such scene graphs can express relational concepts by using sets of triplets ⟨subject—predicate—object⟩. Instead of building dedicated models for scene graph generation, our model tends to extract the latent relational information implicitly encoded in image captioning models. We explored dependency parsing to build grammatically sound parse trees from captions. We used detection algorithms for the region propositions to generate dense region-based concept graphs. These were optimally combined using the approximate sub-graph isomorphism to create holistic concept graphs for images. The major advantages of this approach are threefold. Firstly, the proposed graph generation module is completely rule-based and, hence, adheres to the principles of explainable artificial intelligence. Secondly, graph generation can be used as plug-and-play along with any region proposition and caption generation framework. Finally, our results showed that we could generate rich concept graphs without explicit graph-based supervision.

Toward Holistic Scene Understanding: A Transfer of Human Scene Perception to Mobile Robots

The long-term vision for robotics is to have fully autonomous mobile robots that perceive the environment as humans do or even better. This article transfers the core ideas from human scene perception onto robot scene perception to contribute toward a holistic scene understanding of robots. The first contribution is to extensively survey and compare state-of-the-art robot scene perception approaches with neuroscience theories and studies of human perception. A step-by-step transfer of the perceptual process reveals similarities and differences between robots and humans. The second contribution represents an analysis of the status quo of holistic robot perception approaches to extract to what extent the perceptual capabilities of humans have been reached. Building on this, the gaps and potentials of robot perception are illustrated to address future research directions.

Vision-based holistic scene understanding towards proactive human–robot collaboration

Recently human–robot collaboration (HRC) has emerged as a promising paradigm for mass personalization in manufacturing owing to the potential to fully exploit the strength of human flexibility and robot precision. To achieve better collaboration, robots should be capable of holistically perceiving and parsing the information of a working scene in real-time to plan proactively and act accordingly. Although excessive attentions have been paid to human cognition in existing works of HRC, there is a lack of a holistic consideration of other crucial elements of a working scene, especially when taking a further step towards Proactive HRC. Aiming to fill the gap, this paper provides a systematic review of computer vision-based holistic scene understanding in HRC scenarios, which mainly takes into account the cognition of object, human, and environment along with visual reasoning to gather and compile visual information into semantic knowledge for subsequent robot decision-making and proactive collaboration. Finally, challenges and potential research directions that can be largely facilitated by enhanced holistic perception techniques are also discussed.

Gestalt perception: A game designed to explore players’ gameplay self-efficacy and anxiety reflected in their learning effects

Gestalt perception relates to inferring a holistic scene from separate elements. Using this theory, an application game named Gestalt Puzzle was designed for students to play by recognizing a few parts of an image to reason the whole image of a particular object. Cognitive style can be divided into field independence (FI) and field dependence (FD) depending on individuals’ ability to identify minor components of a whole image. To understand the gameplay learning effect, this study explored how players’ field-independent cognitive style (FI-CS) interacted with their gameplay performance as the two types of cognitive styles were mediated by their self-efficacy and gameplay anxiety in a competition setting. Data from 112 sixth-grade students were collected for confirmatory factor analysis with structural equation modeling. First, participants completed a trial to familiarize themselves with the functions and mechanism of the game; second, all of the participants were divided into teams to compete against each other for 15 minutes by identifying Chinese paintings; they then completed a questionnaire. The results of this study indicated that FI-CS was positively related to gameplay self-efficacy, but negatively related to gameplay anxiety; perceived utility value could be positively predicted by gameplay self-efficacy, but negatively predicted by gameplay anxiety; and perceived value was positively related to game performance. Moreover, FI-CS was positively related to perceived value mediated by gameplay self-efficacy and gameplay anxiety. The implication of this study is that the Gestalt perception game may engage FI-CS learners with higher levels of gameplay self-efficacy, perceived value, and performance, but a lower level of gameplay anxiety. The present study findings imply that teachers can use Gestalt perception games to promote students’ learning of Taiwanese culture.

PSAT-GAN: Efficient Adversarial Attacks Against Holistic Scene Understanding.

Recent advances in deep neural networks (DNNs) have facilitated high-end applications, including holistic scene understanding (HSU), in which many tasks run in parallel with the same visual input. Following this trend, various methods have been proposed to use DNNs to perform multiple vision tasks. However, these methods are task-specific and less effective when considering multiple HSU tasks. End-to-end demonstrations of adversarial examples, which generate one-to-many heterogeneous adversarial examples in parallel from the same input, are scarce. Additionally, one-to-many mapping of adversarial examples for HSU usually requires joint representation learning and flexible constraints on magnitude, which can render the prevalent attack methods ineffective. In this paper, we propose PSAT-GAN, an end-to-end framework that follows the pipeline of HSU. It is based on a mixture of generative models and an adversarial classifier that employs partial weight sharing to learn a one-to-many mapping of adversarial examples in parallel, each of which is effective for its corresponding task in HSU attacks. PSAT-GAN is further enhanced by applying novel adversarial and soft-constraint losses to generate effective perturbations and avoid studying transferability. Experimental results indicate that our method is efficient in generating both universal and image-dependent adversarial examples to fool HSU tasks under either targeted or non-targeted settings.

Event-Centric Hierarchical Representation for Dense Video Captioning

Dense video captioning aims to localize and describe multiple events in untrimmed videos, which is a challenging task that draws attention recently in computer vision. Although existing methods have achieved impressive performance, most of them only focus on local information of event segments or very simple event-level context, overlooking the complexity of event-event relationship and the holistic scene. As a result, the coherence of captions within the same video could be damaged. In this article, we propose a novel event-centric hierarchical representation to alleviate this problem. We enhance the event-level representation by capturing rich relationship between events in terms of both temporal structure and semantic meaning. Then, a caption generator with late fusion is developed to generate surrounding-event-aware and topic-aware sentences, conditioned on the hierarchical representation of visual cues from the scene level, the event level, and the frame level. Furthermore, we propose a duplicate removal method, namely temporal-linguistic non-maximum suppression (TL-NMS) to distinguish redundancy in both localization and captioning stages. Quantitative and qualitative evaluations on the ActivityNet Captions and YouCook2 datasets demonstrate that our method improves the quality of generated captions and achieves state-of-the-art performance on most metrics.

Clouds of Oriented Gradients for 3D Detection of Objects, Surfaces, and Indoor Scene Layouts.

We develop new representations and algorithms for three-dimensional (3D) object detection and spatial layout prediction in cluttered indoor scenes. We first propose a clouds of oriented gradient (COG) descriptor that links the 2D appearance and 3D pose of object categories, and thus accurately models how perspective projection affects perceived image gradients. To better represent the 3D visual styles of large objects and provide contextual cues to improve the detection of small objects, we introduce latent support surfaces. We then propose a "Manhattan voxel" representation which better captures the 3D room layout geometry of common indoor environments. Effective classification rules are learned via a latent structured prediction framework. Contextual relationships among categories and layout are captured via a cascade of classifiers, leading to holistic scene hypotheses that exceed the state-of-the-art on the SUN RGB-D database.

Fusion‐based holistic road scene understanding

This study addresses the problem of holistic road scene understanding based on the integration of visual and range data. To achieve the grand goal, the authors propose an approach that jointly tackles object-level image segmentation and semantic region labelling within a conditional random field (CRF) framework. Specifically, the authors first generate semantic object hypotheses by clustering 3D points, learning their prior appearance models, and using a deep learning method for reasoning their semantic categories. The learned priors, together with spatial and geometric contexts, are incorporated in CRF. With this formulation, visual and range data are fused thoroughly, and moreover, the coupled segmentation and semantic labelling problem can be inferred via graph cuts. The authors’ approach is validated on the challenging KITTI dataset that contains diverse complicated road scenarios. Both quantitative and qualitative evaluations demonstrate its effectiveness.

Enhancing semantic image retrieval with limited labeled examples via deep learning

With the rapid growth of the Internet, a large number of multi-modal objects such as images and their social tags can easily be downloaded from the Web. The use of such objects can improve training process in the presence of few or limited number of labeled images provided. In order to leverage these unlabeled and labeled multi-modal Web objects for enhancing the performance of unimodal image retrieval, we propose a novel approach called Semi-supervised Multi-concept Retrieval to semantic image retrieval via Deep Learning (SMRDL) in this paper. Differing from conventional methods that use multiple and independent concepts in a semantic multi-concept query, our proposed approach regards multiple concepts as a holistic scene for multi-concept scene learning of unimodal retrieval. In particular, we first train a multi-modal Convolutional Neural Network (CNN) as a concept classifier for images and texts, and then use it to annotate unlabeled Web images. For each of unlabeled images, we then obtain its most relevant concept annotations by using a new strategy of annotation promotion. Finally, we employ a unimodal visual CNN to train a concept classifier in visual modality, which uses both unlabeled and labeled examples for concept learning of unimodal retrieval. The results of our comprehensive experiments on two datasets of MIR Flickr 2011 and NUS-WIDE have shown that our proposed approach outperforms several state-of-the-art methods.

Multi-modal multi-concept-based deep neural network for automatic image annotation

Automatic Image Annotation (AIA) remains as a challenge in computer vision with real-world applications, due to the semantic gap between high-level semantic concepts and low-level visual appearances. Contextual tags attached to visual images and context semantics among semantic concepts can provide further semantic information to bridge this gap. In order to effectively capture these semantic correlations, we present a novel approach called Multi-modal Multi-concept-based Deep Neural Network (M2-DNN) in this study, which models the correlations of visual images, contextual tags, and multi-concept semantics. Unlike traditional AIA methods, our M2-DNN approach takes into account not only single-concept context semantics, but also multi-concept context semantics with abstract scenes. In our model, a multi-concept such as $\{``plane",``buildings"\}$ is viewed as one holistic scene concept for concept learning. Specifically, we first construct a multi-modal Deep Neural Network (DNN) as a concept classifier for visual images and contextual tags, and then employ it to annotate unlabeled images. Second, real-world databases commonly include many difficult concepts that are hard to be recognized, such as concepts with similar appearances, concepts with abstract scenes, and rare concepts. To effectively recognize them, we utilize multi-concept semantics inference and multi-modal correlation learning to refine semantic annotations. Finally, we estimate the most relevant labels for each of unlabeled images through a new strategy of label decision. The results of our comprehensive experiments on two publicly available datasets have shown that our method performs favourably compared with several other state-of-the-art methods.

Combining Convolutional Neural Network and Markov Random Field for Semantic Image Retrieval

With the rapidly growing number of images over the Internet, efficient scalable semantic image retrieval becomes increasingly important. This paper presents a novel approach for semantic image retrieval by combining Convolutional Neural Network (CNN) and Markov Random Field (MRF). As a key step, image concept detection, that is, automatically recognizing multiple semantic concepts in an unlabeled image, plays an important role in semantic image retrieval. Unlike previous work that uses single-concept classifiers one by one, we detect semantic multiconcept by using a multiconcept scene classifier. In other words, our approach takes multiple concepts as a holistic scene for multiconcept scene learning. Specifically, we first train a CNN as a concept classifier, which further includes two types of classifiers: a single-concept fully connected classifier that is best suited to single-concept detection and a multiconcept scene fully connected classifier that is good for holistic scene detection. Then we propose an MRF-based late fusion approach that is able to effectively learn the semantic correlation between the single-concept classifier and multiconcept scene classifier. Finally, the semantic correlation among the subconcepts of images is cought to further improve detection precision. In order to investigate the feasibility and effectiveness of our proposed approach, we conduct comprehensive experiments on two publicly available image databases. The results show that our proposed approach outperforms several state-of-the-art approaches.

Large-scale semantic web image retrieval using bimodal deep learning techniques

Semantic web image retrieval is useful to end-users for semantic image searches over the Internet. This paper aims to develop image retrieval techniques for large-scale web image databases. An advanced retrieval system, termed Multi-concept Retrieval using Bimodal Deep Learning (MRBDL), is proposed and implemented using Convolutional Neural Networks (CNNs) which can effectively capture semantic correlations between a visual image and its free contextual tags. Different from existing approaches using multiple and independent concepts in a query, MRBDL considers multiple concepts as a holistic scene for retrieval model learning. In particular, we first use a bimodal CNN to train a holistic scene classifier in two modalities, and then semantic correlations of the sub-concepts included in the images are leveraged to boost holistic scene recognition. The predicted semantic scores obtained from holistic scene classifier are combined with complementary information on web images to improve the retrieval performance. Experiments have been carried out over two publicly available web image databases. The results show that our proposed approach performs favorably compared with several other state-of-the-art methods.

Two-Level Feature Representation for Aerial Scene Classification

Effective scene representation is a fundamental part of high-resolution scene classification systems. In this letter, we present a holistic scene representation method, i.e., the two-level feature representation (TLFR) model. The TLFR is composed of low-level and high-level features. Low-level features are obtained by computing the residual error between a local descriptor and its corresponding visual word, and the high-level features are obtained using a proposed selection-constrained sparse coding method. In addition, low-level features in a cluster are integrated by summation pooling, whereas high-level features are fused by maximization pooling. The holistic scene representation is finally generated by incorporating these two levels of features into the bag-of-visual-words framework. Experimental results show that the TLFR model is robust to translation and rotation variations and demonstrates promising performance with the Land Use and Land Cover Database data set and a newly released Singapore data set.

Automatic non-parametric image parsing via hierarchical semantic voting based on sparse–dense reconstruction and spatial–contextual cues

Image parsing is vital for many high-level image understanding tasks. Although both parametric and non-parametric approaches have achieved remarkable success, many technical challenges still prevail for images containing things/objects with broad-coverage and high-variability, because it still lacks versatile and effective strategies to seamlessly integrate local–global features selection, contextual cues exploitation, spatial layout encoding, data-driven coherency exploration, and flexible accommodation of newly annotated labels. To ameliorate, this paper develops a novel automatic non-parametric image parsing method with advantages of both parametric and non-parametric methodologies by resorting to new modeling and inferring strategies. The originality of our new approach is to employ sparse–dense reconstruction as a latent learning model to conduct candidate-label probability analysis over multi-level local regions, and synchronously leverage context-specific local–global label confidence propagation and global semantic spatial–contextual cues to guide holistic scene parsing. Towards this goal, we devise several novel technical components to comprise a lightweight parsing framework, including local region representation integrating complementary features, anisotropic consistency propagation based on bi-harmonic distance metric, bottom-up label voting, semantic string generation of image-level spatial–contextual cues based on Hilbert space-filling curve, and co-occurrence priors analysis based on relaxed string matching algorithm, which collectively enable us to effectively combat the aforementioned obstinate problems. Moreover, we conduct comprehensive experiments on public benchmarks, and make extensive and quantitative evaluations with state-of-the-art methods, which demonstrate the advantages of our method in accuracy, versatility, flexibility, and efficiency.