Exploiting Context Information Research Articles

Deep image compression based on multi-scale deformable convolution

Deep image compression efficiency has been improved in the past years. However, to fully exploit context information for compressing image objects of different scales and shapes, more adaptive geometric structure of inputs should be considered. In this paper, we novelly introduce deformable convolution and its spatial attention extension into deep image compression task to fully exploit the context information. Specifically, a novel deep image compression network with Multi-Scale Deformable Convolution and Spatial Attention, named MS-DCSA, is proposed to better extract compact and efficient latent representation as well as reconstruct higher-quality images. First, multi-scale deformable convolution is presented to provide multi-scale receptive fields for learning spatial sampling offsets in deformable operations. Subsequently, multi-scale deformable spatial attention module is developed to generate attention masks to re-weight extracted features according to their importance. In addition, the multi-scale deformable convolution is applied to design delicate up/down sampling modules. Extensive experiments demonstrate that the proposed MS-DCSA network achieves improved performance on both PSNR and MS-SSIM quality metrics, compared to conventional as well as competing deep image compression methods.

ASF-Net: Adaptive Screening Feature Network for Building Footprint Extraction From Remote-Sensing Images

Building footprint extraction plays an important role in many remote-sensing (RS) applications such as urban planning and disaster monitoring. Mainly, the exploitation of contextual information in a fixed receptive field is the focus of previous research, which makes it difficult to generically extract buildings that vary greatly in size and shape, especially when isolated large buildings are surrounded by dense small buildings. To improve this problem, we attempt to teach the network to adjust the receptive field and enhance useful feature information adaptively. In this article, we propose a novel adaptive screening feature network (ASF-Net), which can independently screen and enhance effective feature information from two aspects. On the one hand, we propose a deepened space up-sampling block to screen useful information and help establish boundaries. On the other hand, we propose an Adaptive Information Utilization Block (AIUB) to enlarge the receptive field of feature maps and refine the incomplete building footprint. As a result, the more accurate multiscale building footprint is inferred from the enhanced features. Experimental results on the popular aerial image segmentation datasets show that ASF-Net obtains competitive results [80.2% intersection over union (IoU) on the Inria aerial image labeling dataset and 74.2% IoU on the Massachusetts buildings dataset] in comparison with several state-of-the-art models. The TensorFlow implementation is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/jyx0516/ASF-Net</uri> .

Using Context-Awareness for Storage Services in Edge Computing

Modern mobile networks face a dynamic environment with massive devices and heterogeneous service expectations that will need to significantly scale for 5G. Edge computing approaches aim at enhancing scalability through strategies like computation offloading and local storage services, which will be fundamental to deploying large-scale distributed applications. Unlike the cloud, edge resources are limited, which call for novel techniques to handle large volumes of up- and downstream data under a changing environment. Being closer to data consumers and producers, a compelling view is to adopt context-aware techniques for enabling the edge to work with patterns from mobile traffic at different spatiotemporal scales. In this article, we overview the challenges and opportunities of edge storage from the perspective of context-awareness. We introduce a conceptual architecture to learn and exploit context information for enhancing uplink and downlink scenarios. Finally, we outline future directions for edge applications.

Realize your surroundings: Exploiting context information for small object detection

Small object detection is a highly challenging problem due to the limited resolution and information of small objects. Current state-of-the-art detectors only utilize the appearance feature to locate and classify objects. However, such detectors are prone to failure when detecting small objects, especially in the case of heavy appearance changes and background distractors, in which the appearance feature alone is not sufficient for robust detection. Exploiting context information in the surrounding scene can be highly beneficial in such cases. In this paper, we propose a novel detector, the Internal-External Network (IENet), which uses both the appearance and context information of the object for robust detection. In the proposed approach, small object detection is improved from feature extraction, proposal location, and classification. Specifically, three customized modules are designed, including the Bidirectional Feature Fusion Module (Bi-FFM), Context Reasoning Module (CRM), and Context Feature Augmentation Module (CFAM). Bi-FFM is designed to capture the internal feature of objects by transferring the semantic feature of deeper-level layers to lower-level layers and the detailed feature of lower-level layers to deeper-level layers in neural networks. The proposed approach not only utilizes the hierarchy of convolutional features but also improves its prediction via context relationships. CRM is designed to improve the quality of region proposals by context reasoning that uses easily detected objects to help understand hard ones. Furthermore, CFAM is designed to learn pair-wise relations between region proposals produced by CRM, and such relations are used to produce global feature information associated with the region proposals for accurate classification. Extensive experiments are conducted on the challenging COCO and WIDER FACE datasets to demonstrate the effectiveness of the proposed approach. Experimental results show that the detection performance of small objects is greatly improved over state-of-the-art detectors.

SemiText: Scene text detection with semi-supervised learning

Scene text detection is an important step of scene text recognition and has achieved significant progress. However, the requirement of large amounts of annotated training data, which is used for training text detection model, has become a great challenge for existing methods. In this paper, we propose a semi-supervised scene text detection framework (SemiText), which trains robust and accurate scene text detectors using a pre-trained supervised model and the unannotated data. With a pre-trained model that is pre-trained on the fully annotated synthetic dataset, i.e., SynthText, we investigate the inductive and transductive semi-supervised learning on the unannotated dataset respectively. For inductive learning, the pre-trained model is applied to the unannotated training dataset to search for more training examples, which are further combined with SynthText to fine-tune the pre-trained model and achieve a superior detection model. For transductive learning, the unannotated training dataset is replaced with the unannotated test dataset. Meanwhile, for the aim of real-world applications, we adopt Mask R-CNN to detect text with arbitrary shapes and exploit context information to suppress false positives. Extensive experiments on different datasets show that the performance of our text detection method can be clearly improved under both inductive and transductive semi-supervision. Additionally, we also achieve state-of-the-art performance under full supervision.

A Comparative Study for Unsupervised Network Representation Learning

There has been appreciable progress in unsupervised network representation learning (UNRL) approaches over graphs recently with flexible random-walk approaches, new optimization objectives and deep architectures. However, there is no common ground for systematic comparison of embeddings to understand their behavior for different graphs and tasks. In this paper we theoretically group different approaches under a unifying framework and empirically investigate the effectiveness of different network representation methods. In particular, we argue that most of the UNRL approaches either explicitly or implicit model and exploit context information of a node. Consequently, we propose a framework that casts a variety of approaches -- random walk based, matrix factorization and deep learning based -- into a unified context-based optimization function. We systematically group the methods based on their similarities and differences. We study the differences among these methods in detail which we later use to explain their performance differences (on downstream tasks). We conduct a large-scale empirical study considering 9 popular and recent UNRL techniques and 11 real-world datasets with varying structural properties and two common tasks -- node classification and link prediction. We find that there is no single method that is a clear winner and that the choice of a suitable method is dictated by certain properties of the embedding methods, task and structural properties of the underlying graph. In addition we also report the common pitfalls in evaluation of UNRL methods and come up with suggestions for experimental design and interpretation of results.

An Object Context Integrated Network for Joint Learning of Depth and Optical Flow

Supervised depth prediction and optical flow estimation have achieved promising performance due to the advanced deep network architectures. Since the ground truths are difficult to be collected, many recent works try to learn the depth and flow in an unsupervised manner. However, existing methods only use features from convolutional layers or a simple aggregation of multi-level features to predict the depth and flow maps, which is insufficient to exploit context information. In this paper, we attempt to exploit object contextual information and investigate the effect of the object context for joint learning of depth and optical flow. Specifically, we present a novel combination of object context and the framework of joint learning depth and optical flow. Our proposed network can exploit and integrate the object context for both tasks by aggregating the context according to pair-wise similarities. Furthermore, we adopt the existing spatial pyramid network (SPN) to estimate the depth and flow in a coarse-to-fine strategy effectively. Given temporally adjacent stereo pairs, our network can be trained end-to-end in an unsupervised manner and can predict the depth and flow maps simultaneously. We conduct experiments on two publicly available datasets, KITTI2012 and KITTI2015. Our proposed approach yields comparable performance on both depth and flow tasks, compared to the recent deep learning-based approaches. Experimental results demonstrate that exploiting object contextual information is useful and beneficial for depth and optical flow estimation.

Self-Supervised deep homography estimation with invertibility constraints

Remarkable performance of the homography estimation has been achieved by the deep CNN based approaches. These homography estimation methods, more often than not, are supervised methods and rely too much on the ground truth annotations as they aim to learn the mapping between image pairs and homography. On the other hand, the inherent invertibility of homography is helpful to avoid over-fitting and improve the performance, which however is ignored by previous homography estimation methods. In this paper, we propose a novel homography estimation approach, named “Self-Supervised Regression Network(SSR-Net)”, which relaxes the need of ground truth annotations and takes advantage of invertibility constraints. We utilize spatial pyramid pooling modules to improve the quality of extracted features in each image by exploiting context information. To employ the invertibility constraints, we adopt the matrix representation of the homography rather than the commonly used 4-point parameterization in other methods. Our proposed SSR-Net produce homography matrices and synthetic images in a cycled way. The network are trained in a self-supervised way by minimizing the combination of photometric loss and invertibility loss. Experiments on the synthetic dataset generated from MSCOCO dataset show that our proposed method outperforms several state-of-the-art approaches.

Exploiting context based on CNN and coding representations for pedestrian co-detection

The exploitation of contextual information among multiple images has been proven significant to improve detection performance by object co-detection methods. In this paper, we propose a pedestrian co-detection method that combines the strengths of convolutional neural networks (CNNs) and locality-constrained linear coding (LLC) in a unified conditional random field (CRF) model. First, we obtain object candidates by using a region proposal network (RPN) in Faster R-CNN. Second, we build a fully connected CRF that consists of unary potentials on individual object candidates and two types of pairwise potentials on pairs of object candidates. The unary potential is computed independently for each object candidate by using the baseline method. The pairwise potentials consist of multiscale CNN and LLC representation-based potentials, which contribute to the capturing of relationships among object candidates in all the test images. Finally, we jointly predict the category labels of all the object candidates through the mean field inference in the CRF. We evaluated the proposed method on the ETH, Caltech, and INRIA Pedestrian datasets. The experimental results demonstrate the effectiveness of the proposed method as compared to the baseline method.

Context multi-task visual object tracking via guided filter

In this paper, we formulate particle filter based tracking as a multi-task sparse learning problem that exploits context information. The target and context information which modeled as linear combinations of principal component analysis (PCA) basis is formed as dictionary templates. The target is treated as the guidance and the sampling examples are filtered depending on the similarity between the target and each input. The edge preserving smoothing property of the guided filter is a key factor for object tracking. First, valuable candidates can be selected and the inaccurate candidates become blurry. Therefore, the guided filter can help to distinguish the target from numerous candidates easily. Second, partial occluded target can be recovered by the filtering process via the guidance image. Thus, the drifting problem can be alleviated. Then multi-task sparse learning is employed to learn the target and context information. The proposed learning problem is efficiently solved using an alternating direction method of multipliers (ADMM) method that yields a sequence of closed form updates. We test our tracker on challenging video sequences that involve drastic illumination changes, large pose variations, and heavy occlusions. Experimental results show that our tracker consistently outperforms state-of-the-art trackers.

Characterizing context-aware recommender systems: A systematic literature review

Context-aware recommender systems leverage the value of recommendations by exploiting context information that affects user preferences and situations, with the goal of recommending items that are really relevant to changing user needs. Despite the importance of context-awareness in the recommender systems realm, researchers and practitioners lack guides that help them understand the state of the art and how to exploit context information to smarten up recommender systems. This paper presents the results of a comprehensive systematic literature review we conducted to survey context-aware recommenders and their mechanisms to exploit context information. The main contribution of this paper is a framework that characterizes context-aware recommendation processes in terms of: i) the recommendation techniques used at every stage of the process, ii) the techniques used to incorporate context, and iii) the stages of the process where context is integrated into the system. This systematic literature review provides a clear understanding about the integration of context into recommender systems, including context types more frequently used in the different application domains and validation mechanisms—explained in terms of the used datasets, properties, metrics, and evaluation protocols. The paper concludes with a set of research opportunities in this field.

See It With Your Own Eyes: Markerless Mobile Augmented Reality for Radiation Awareness in the Hybrid Room.

We present an approach to provide awareness to the harmful ionizing radiation generated during X-ray-guided minimally invasive procedures. A hand-held screen is used to display directly in the user's view information related to radiation safety in a mobile augmented reality (AR) manner. Instead of using markers, we propose a method to track the observer's viewpoint, which relies on the use of multiple RGB-D sensors and combines equipment detection for tracking initialization with a KinectFusion-like approach for frame-to-frame tracking. Two of the sensors are ceiling-mounted and a third one is attached to the hand-held screen. The ceiling cameras keep an updated model of the room's layout, which is used to exploit context information and improve the relocalization procedure. The system is evaluated on a multicamera dataset generated inside an operating room (OR) and containing ground-truth poses of the AR display. This dataset includes a wide variety of sequences with different scene configurations, occlusions, motion in the scene, and abrupt viewpoint changes. Qualitative results illustrating the different AR visualization modes for radiation awareness provided by the system are also presented. Our approach allows the user to benefit from a large AR visualization area and permits to recover from tracking failure caused by vast motion or changes in the scene just by looking at a piece of equipment. The system enables the user to see the 3-D propagation of radiation, the medical staff's exposure, and/or the doses deposited on the patient's surface as seen through his own eyes.

Context-Aware Handover Policies in HetNets

Next generation cellular systems are expected to entail a wide variety of wireless coverage zones, with cells of different sizes and capacities that can overlap in space and share the transmission resources. In this scenario, which is referred to as Heterogeneous Networks (HetNets), a fundamental challenge is the management of the handover process between macro, femto and pico cells. To limit the number of handovers and the signaling between the cells, it will hence be crucial to manage the user’s mobility considering the context parameters, such as cells size, traffic loads, and user velocity. In this paper, we propose a theoretical model to characterize the performance of a mobile user in a HetNet scenario as a function of the user’s mobility, the power profile of the neighboring cells, the handover parameters, and the traffic load of the different cells. We propose a Markov-based framework to model the handover process for the mobile user, and derive an optimal context-dependent handover criterion. The mathematical model is validated by means of simulations, comparing the performance of our strategy with conventional handover optimization techniques in different scenarios. Finally, we show the impact of the handover regulation on the users performance and how it is possible to improve the users capacity exploiting context information.

Context-aware opportunistic networking in multi-hop cellular networks

5G networks will be required to efficiently support the growth in mobile data traffic. One approach to do so is by exploiting Device-to-Device (D2D) communications and Multi-Hop Cellular Networks (MCNs) in order to enhance the spectrum re-use and offload traffic over underlay networks. This study proposes to further improve the efficiency of transmitting mobile data traffic by integrating opportunistic networking principles into MCNs. Opportunistic networking can exploit the delay tolerance characteristic of relevant data traffic services in order to search for the most efficient transmission conditions in MCNs. The study first presents an analytical framework for two-hop opportunistic MCNs designed to identify their optimum configuration in terms of energy efficiency. Using this reference configuration, the paper then proposes a set of opportunistic forwarding policies that exploit context information provided by the cellular network. Numerical and simulation results demonstrate that opportunistic networking can significantly contribute towards achieving the capacity and energy efficiency gains sought for 5G networks. Under the evaluated conditions, the obtained results show that the proposed schemes can reduce the energy consumption compared to traditional cellular communications by up to 98% for delay tolerant services. In addition, the proposed schemes can increase the cellular capacity by up to 79% compared to traditional cellular communications.

CRF learning with CNN features for image segmentation

Conditional Random Rields (CRF) have been widely applied in image segmentations. While most studies rely on hand-crafted features, we here propose to exploit a pre-trained large convolutional neural network (CNN) to generate deep features for CRF learning. The deep CNN is trained on the ImageNet dataset and transferred to image segmentations here for constructing potentials of superpixels. Then the CRF parameters are learnt using a structured support vector machine (SSVM). To fully exploit context information in inference, we construct spatially related co-occurrence pairwise potentials and incorporate them into the energy function. This prefers labelling of object pairs that frequently co-occur in a certain spatial layout and at the same time avoids implausible labellings during the inference. Extensive experiments on binary and multi-class segmentation benchmarks demonstrate the promise of the proposed method. We thus provide new baselines for the segmentation performance on the Weizmann horse, Graz-02, MSRC-21, Stanford Background and PASCAL VOC 2011 datasets.

An Analysis of Object Appearance Information and Context Based Classification

The purpose of categorizing objects is to identify and locate within an image, instances of an object category. It is a difficult task to identify objects in images when such images contain poor quality, occlusion, background clutter, or noise. It is more difficult when there are too many objects in same scene (background). Most models for categorizing object exploit context information and appearance information from objects in order to enhance recognition accuracy. In the scene, interaction among objects (context information) can successfully help in removing ambiguity of appearance inputs during recognition task. Appearance information can help identify object classes successfully up to some level. The issue of integrating several kinds of contextual information in order to have a robust categorization of object is address in this paper. Various methods of exploiting contextual information for categorizing object are review. An isolated object. The object’s appearance is not sufficient enough to reveal the accurate information about the identity of the object.

Energy‐efficient opportunistic forwarding in multi‐hop cellular networks using device‐to‐device communications

AbstractCellular networks face significant capacity, efficiency and quality challenges because of the exponential growth of cellular data traffic. Multi‐hop cellular networks (MCNs) have been proposed to address these challenges through the integration of cellular and device‐to‐device communications. This work investigates how the adoption and design of opportunistic store, carry and forward mechanisms in MCNs can help importantly decrease the energy consumption for delay tolerant traffic. To this aim, the study first derives an analytical framework to identify in two‐hop scenarios the optimum mobile relay location and the location from which the mobile relay should start forwarding the information to the cellular base station in order to minimise the overall energy consumption. The derived optimum configuration is then used as a benchmark for the proposal and design of a novel context‐based opportunistic MCN forwarding scheme that exploits context information provided at low cost by the cellular network. Numerical and simulation results demonstrate that the proposed context‐based opportunistic forwarding can achieve a performance close to that obtain with the optimum configuration and reduce the energy consumption by more than 90% compared with traditional single‐hop cellular communications. Copyright © 2014 John Wiley &amp; Sons, Ltd.

Robust Detection of Abandoned and Removed Objects in Complex Surveillance Videos

Tracking-based approaches for abandoned object detection often become unreliable in complex surveillance videos due to occlusions, lighting changes, and other factors. We present a new framework to robustly and efficiently detect abandoned and removed objects based on background subtraction (BGS) and foreground analysis with complement of tracking to reduce false positives. In our system, the background is modeled by three Gaussian mixtures. In order to handle complex situations, several improvements are implemented for shadow removal, quick-lighting change adaptation, fragment reduction, and keeping a stable update rate for video streams with different frame rates. Then, the same Gaussian mixture models used for BGS are employed to detect static foreground regions without extra computation cost. Furthermore, the types of the static regions (abandoned or removed) are determined by using a method that exploits context information about the foreground masks, which significantly outperforms previous edge-based techniques. Based on the type of the static regions and user-defined parameters (e.g., object size and abandoned time), a matching method is proposed to detect abandoned and removed objects. A person-detection process is also integrated to distinguish static objects from stationary people. The robustness and efficiency of the proposed method is tested on IBM Smart Surveillance Solutions for public safety applications in big cities and evaluated by several public databases, such as The Image library for intelligent detection systems (i-LIDS) and IEEE Performance Evaluation of Tracking and Surveillance Workshop (PETS) 2006 datasets. The test and evaluation demonstrate our method is efficient to run in real-time, while being robust to quick-lighting changes and occlusions in complex environments.

Special issue on “context-aware data mining (CADM)”

Knowledge discovery is generally described as the process of automatic extraction of interesting and previously unknown patterns from large amounts of data. However, intelligent methods that explicitly incorporate semantic information into the discovery process are not yet well established in the literature. The exploitation of context during the mining process is one of the most common techniques to take advantage of the semantics of different application domains. Contextual aspects may intervene in several steps of the discovery process, but the question is how these aspects interact within the process, and the forms that this can take. For instance, in data pre-processing, enriching the data with additional context information can lead to the generation of more meaningful and human understandable patterns, once the generated patterns are directly related to the input data. Also, during the mining task, context may be used as constraints, thus allowing search space reduction, pattern pruning, and the development of more efficient algorithms. Context information and domain knowledge may intervene in the post-processing steps, helping in the explanation of the results. Recent years have witnessed increasing interests in exploiting context information in data mining. Also, data mining has been successfully applied in a number of different fields, such as bioinformatics, business intelligence, finance data analysis, location based services, network data analysis, and social link analysis. This special issue is an example of this, since the selected papers are related to different data mining areas such as privacy, sensor data, text mining, graph mining, and ontologies. As a consequence, context-aware data mining is a never-ending resilience field and attracts growing research efforts from different fields.

Ontology-based context representation and reasoning for object tracking and scene interpretation in video

Computer vision research has been traditionally focused on the development of quantitative techniques to calculate the properties and relations of the entities appearing in a video sequence. Most object tracking methods are based on statistical methods, which often result inadequate to process complex scenarios. Recently, new techniques based on the exploitation of contextual information have been proposed to overcome the problems that these classical approaches do not solve. The present paper is a contribution in this direction: we propose a Computer Vision framework aimed at the construction of a symbolic model of the scene by integrating tracking data and contextual information. The scene model, represented with formal ontologies, supports the execution of reasoning procedures in order to: (i) obtain a high-level interpretation of the scenario; (ii) provide feedback to the low-level tracking procedure to improve its accuracy and performance. The paper describes the layered architecture of the framework and the structure of the knowledge model, which have been designed in compliance with the JDL model for Information Fusion. We also explain how deductive and abductive reasoning is performed within the model to accomplish scene interpretation and tracking improvement. To show the advantages of our approach, we develop an example of the use of the framework in a video-surveillance application.