Indoor Semantic Research Articles

Indoor Semantic Location Privacy Protection With Safe Reinforcement Learning

Indoor Semantic Location Privacy Protection With Safe Reinforcement Learning

NGLSFusion: Non-Use GPU Lightweight Indoor Semantic SLAM

Perception of the indoor environment is the basis of mobile robot localization, navigation, and path planning, and it is particularly important to construct semantic maps in real time using minimal resources. The existing methods are too dependent on the graphics processing unit (GPU) for acquiring semantic information about the indoor environment, and cannot build the semantic map in real time on the central processing unit (CPU). To address the above problems, this paper proposes a non-use GPU for lightweight indoor semantic map construction algorithm, named NGLSFusion. In the VO method, ORB features are used for the initialization of the first frame, new keyframes are created by optical flow method, and feature points are extracted by direct method, which speeds up the tracking speed. In the semantic map construction method, a pretrained model of the lightweight network LinkNet is optimized to provide semantic information in real time on devices with limited computing power, and a semantic point cloud is fused using OctoMap and Voxblox. Experimental results show that the algorithm in this paper ensures the accuracy of camera pose while speeding up the tracking speed, and obtains a reconstructed semantic map with complete structure without using GPU.

Crowdsourcing-Based Indoor Semantic Map Construction and Localization Using Graph Optimization.

The advancement of smartphones with multiple built-in sensors facilitates the development of crowdsourcing-based indoor map construction and localization. This paper proposes a crowdsourcing-based indoor semantic map construction and localization method using graph optimization. Using waypoints, semantic landmarks, and Wi-Fi landmarks as nodes and the relevance between waypoints and landmarks (i.e., waypoint–waypoint, waypoint–semantic, waypoint–Wi-Fi, semantic–semantic, and Wi-Fi–Wi-Fi) as edges, the optimization graph is constructed. Initializing the venue map is the single-track semantic map with the highest quality, as determined by a proposed map quality evaluation function. The aligned venue and candidate maps are optimized while satisfying the constraints, with the candidate map exhibiting the highest degree of similarity to the venue map. The lightweight venue map is then updated in terms of waypoint and landmark attributes, as well as the relationship between waypoints and landmarks. To determine a pedestrian’s location on a venue map, similarities between a local map and a venue map are evaluated. Experiments conducted in an office building and shopping mall scenes demonstrate that crowdsourcing-based venue maps are superior to single-track semantic maps. Additionally, the landmark matching-based localization method can achieve a mean localization error of less than 0.5 m on the venue map, compared to 0.6 m in a single-track semantic map.

A New Approach to Measuring the Similarity of Indoor Semantic Trajectories

People spend more than 80% of their time in indoor spaces, such as shopping malls and office buildings. Indoor trajectories collected by indoor positioning devices, such as WiFi and Bluetooth devices, can reflect human movement behaviors in indoor spaces. Insightful indoor movement patterns can be discovered from indoor trajectories using various clustering methods. These methods are based on a measure that reflects the degree of similarity between indoor trajectories. Researchers have proposed many trajectory similarity measures. However, existing trajectory similarity measures ignore the indoor movement constraints imposed by the indoor space and the characteristics of indoor positioning sensors, which leads to an inaccurate measure of indoor trajectory similarity. Additionally, most of these works focus on the spatial and temporal dimensions of trajectories and pay less attention to indoor semantic information. Integrating indoor semantic information such as the indoor point of interest into the indoor trajectory similarity measurement is beneficial to discovering pedestrians having similar intentions. In this paper, we propose an accurate and reasonable indoor trajectory similarity measure called the indoor semantic trajectory similarity measure (ISTSM), which considers the features of indoor trajectories and indoor semantic information simultaneously. The ISTSM is modified from the edit distance that is a measure of the distance between string sequences. The key component of the ISTSM is an indoor navigation graph that is transformed from an indoor floor plan representing the indoor space for computing accurate indoor walking distances. The indoor walking distances and indoor semantic information are fused into the edit distance seamlessly. The ISTSM is evaluated using a synthetic dataset and real dataset for a shopping mall. The experiment with the synthetic dataset reveals that the ISTSM is more accurate and reasonable than three other popular trajectory similarities, namely the longest common subsequence (LCSS), edit distance on real sequence (EDR), and the multidimensional similarity measure (MSM). The case study of a shopping mall shows that the ISTSM effectively reveals customer movement patterns of indoor customers.

TSNet: Three-Stream Self-Attention Network for RGB-D Indoor Semantic Segmentation

This article proposes a three-stream self-attention network (TSNet) for indoor semantic segmentation comprising two asymmetric input streams (asymmetric encoder structure) and a cross-modal distillation stream with a self-attention module. The two asymmetric input streams are ResNet34 for the red-green-blue (RGB) stream and VGGNet16 for the depth stream. Accompanying the RGB and depth streams, a cross-modal distillation stream with a self-attention module extracts new RGB plus depth features in each level in the bottom-up path. In addition, while using bilinear upsampling to recover the spatial resolution of the feature map, we incorporated the feature information of both the RGB flow and the depth flow through the self-attention module. We constructed the NYU Depth V2 dataset to evaluate the TSNet and achieved results comparable to those of current state-of-the-art methods.

Indoor Semantic Mapping with Efficient Convolutional Neural Networks for Resource-constrained SLAM System

Scene parsing and robotic manipulation using visual sensing are of significant importance for mobile and robotics related applications. And increasing efforts are being made to associate semantic concepts to geometric entities in robot’s surroundings. However, they are not well suitable for resource-constrained and mobile platforms. Toward an efficient semantic mapping solution, we propose a specifically light-weight segmentation network for building a dense 3D semantic map in real-time. The propose architecture is based on depth-wise convolution and channel shuffle to reduce computation cost. An improved Atrous Spatial Pyramid Pooling (ASPP) and encoder-decoder structures are presented and multi-scale features are merged for better prediction. Experiments are compared with other methods to prove the effectiveness and superiority.

Finding Most Popular Indoor Semantic Locations Using Uncertain Mobility Data

Knowing popular indoor locations can benefit many applications like exhibition planning and location-based advertising, among others. In this work, we use uncertain historical indoor mobility data to find the top-$k$k popular indoor semantic locations with the highest flow values. In the data we use, an object positioning report contains a set of samples, each consisting of an indoor location and a corresponding probability. The problem is challenging due to the difficulty in obtaining reliable flow values and the heavy computational workload on probabilistic samples for large numbers of objects. To address the first challenge, we propose an indoor flow definition that takes into account both data uncertainty and indoor topology. To efficiently compute flows for individual indoor semantic locations, we design data structures for facilitating accessing the relevant data, a data reduction method that reduces the intermediate data to process, and an overall flow computing algorithm. Furthermore, we design search algorithms for finding the top-$k$k popular indoor semantic locations. All proposals are evaluated extensively on real and synthetic data. The evaluation results show that our data reduction method significantly reduces the data volume in computing, our search algorithms are efficient and scalable, and the top-$k$k popular semantic locations returned are in good accord with ground truth.

WiFiMap+: High-Level Indoor Semantic Inference With WiFi Human Activity and Environment

Existing indoor semantic recognition schemes are capable of discovering patterns through smartphone sensing; however, there is a lack of a device-free indoor semantic recognition system. In this paper, we propose WiFiMap+, which is a first-ever automatical inference system using WiFi signals to recognize high-level indoor semantics from human activities and environments, where the high-level indoor semantics consist of indoor facilities and environments. To characterize the static indoor environments and dynamical human activities separately with channel state information (CSI), we propose a novel two-stream architecture to generate the spatial streams and the movement streams independently. Compared to the recent research on activity recognition, this two-stream architecture can make the content area of CSI samples extend from human activities to indoor environments. For obtaining accurate indoor environment detection, we propose a CSI-environment model with a spatial stream generation algorithm, which can reduce the effect of human activities on environment detection. For stable activity recognition, we also propose an environment-based testing sample representation method, which can utilize the environment knowledge to overcome the diversity of CSI caused by the environment changes. Finally, we implement WiFiMap+ using commercial WiFi devices and evaluate its performance for seven common semantic detection cases in six-room scenarios. The experimental results show that our proposed WiFiMap+ is robust to the multi-room scenario and can achieve the average accuracy of $\text{92.8}\%$ and the lowest accuracy of about $\text{82}\%$ .

SISE: Self-Updating of Indoor Semantic Floorplans for General Entities

Indoor semantic floorplan is important for a range of location based service (LBS) applications, attracting many research efforts in several years. In many cases, the out-of-date indoor semantic floorplans would gradually deteriorate and even break down the LBS performance. Thus, it is important to automatically update changed semantics of indoor floorplans caused by environmental variation. However, few research has been focused on the continuous semantic updating problem. This paper presents SISE as a mobile crowdsourcing system that uses a new abstraction for indoor general entities and their semantics, enGraph, to automatically update changed semantics of indoor floorplans using images and inertial data. We first propose efficient methods to generate enGraph. Thus, an image can be associated with an indoor semantic floorplan. Accordingly, we formulate the enGraph matching problem and then propose a quality-based maximum common subgraph matching algorithm so that entities extracted from an image can be corresponded to entities in the indoor semantic floorplan. Furthermore, we propose a quadrant comparison algorithm and a region shrink based localization algorithm to detect and localize changed entities. Thus, the new semantics can be labeled and out-of-date semantics can be removed. Extensive experiments have been conducted on real and synthetic data. Experimental results show that 80 percent of out-of-date semantics of indoor general entities can be updated by SISE.

Indoor Semantic Modelling for Routing: The Two-Level Routing Approach for Indoor Navigation

Humans perform many activities indoors and they show a growing need for indoor navigation, especially in unfamiliar buildings such as airports, museums and hospitals. Complexity of such buildings poses many challenges for building managers and visitors. Indoor navigation services play an important role in supporting these indoor activities. Indoor navigation covers extensive topics such as: 1) indoor positioning and localization; 2) indoor space representation for navigation model generation; 3) indoor routing computation; 4) human wayfinding behaviours; and 5) indoor guidance (e.g., textual directories). So far, a large number of studies of pedestrian indoor navigation have presented diverse navigation models and routing algorithms/methods. However, the major challenge is rarely referred to: how to represent the complex indoor environment for pedestrians and conduct routing according to the different roles and sizes of users. Such complex buildings contain irregular shapes, large open spaces, complicated obstacles and different types of passages. A navigation model can be very complicated if the indoors are accurately represented. Although most research demonstrates feasible indoor navigation models and related routing methods in regular buildings, the focus is still on a general navigation model for pedestrians who are simplified as circles. In fact, pedestrians represent different sizes, motion abilities and preferences (e.g., described in user profiles), which should be reflected in navigation models and be considered for indoor routing (e.g., relevant Spaces of Interest and Points of Interest)...

DeepMap+: Recognizing High-Level Indoor Semantics Using Virtual Features and Samples Based on a Multi-Length Window Framework.

Existing indoor semantic recognition schemes are mostly capable of discovering patterns through smartphone sensing, but it is hard to recognize rich enough high-level indoor semantics for map enhancement. In this work we present DeepMap+, an automatical inference system for recognizing high-level indoor semantics using complex human activities with wrist-worn sensing. DeepMap+ is the first deep computation system using deep learning (DL) based on a multi-length window framework to enrich the data source. Furthermore, we propose novel methods of increasing virtual features and virtual samples for DeepMap+ to better discover hidden patterns of complex hand gestures. We have performed 23 high-level indoor semantics (including public facilities and functional zones) and collected wrist-worn data at a Wal-Mart supermarket. The experimental results show that our proposed methods can effectively improve the classification accuracy.

Indoor Semantic Modelling for Routing: The Two-Level Routing Approach for Indoor Navigation

Indoor Semantic Modelling for Routing: The Two-Level Routing Approach for Indoor Navigation

From mapping to indoor semantic queries: Enabling zero-effort indoor environmental sensing

Understanding indoor environment in an automatic way is of great importance to mobile and pervasive computing. In this paper, we present Zeus, a smartphone-based opportunistic sensing system that automatically constructs indoor maps by merging crowdsourced walking trajectories captured through smartphone inertial sensing. Most importantly, widely used indoor semantics, such as stairs, escalators, elevators and doors, are also automatically detected and annotated to the constructed maps in the same inference process. Since the final inferred maps provide locations of the different indoor semantics together with localization database, Zeus enables real-time location-based semantic queries. The evaluation result shows that Zeus accurately infers semantic-annotated indoor maps, and provide accurate semantic query in different indoor environments.

Local N-ary Patterns: a local multi-modal descriptor for place categorization

This paper presents an effective integration method of multiple modalities such as depth, color, and reflectance for place categorization. To achieve better performance with integrated multi-modalities, we introduce a novel descriptor, local N-ary patterns (LTP), which can perform robust discrimination of place categorization. In this paper, the LNP descriptor is applied to a combination of two modalities, i.e. depth and reflectance, provided by a laser range finder. However, the LNP descriptor can be easily extended to a larger number of modalities. The proposed LNP describes relationships between the multi-modal values of pixels and their neighboring pixels. Since we consider the multi-modal relationship, our proposed method clearly demonstrates more effective classification results than using individual modalities. We carried out experiments with the Kyushu University Indoor Semantic Place Dataset, which is publicly available. This data-set is composed of five indoor categories: corridors, kitchens, laboratories, study rooms, and offices. We confirmed that our proposed method outperforms previous uni-modal descriptors.

Indoor Semantic Mapping for Mobile Robot

Indoor Semantic Mapping for Mobile Robot

Textual Signs Reading for Indoor Semantic Map Construction

Semantic maps have come up to facilitate high-level human robot interaction like “deliver a laptop for maintenance” . Textual signs posted on the walls and room doors in indoor environments are environment features that could be easily identified by service robots. These signs provide some semantic information like the function of the room and its occupants. In addition, they consider landmarks for the service robots. This paper is a continuous work of developing a framework for creating a semantic map for indoor environment using mobile robot [17]. The paper addresses the problem of automatically detecting and recognizing textual signs during robotic mapping. Then, annotating these recognized signs to a previously generated robotic grid map. ATRV-mini robot has been used in our experiments.