Spatial Relationship Features Research Articles

Fine-grained image classification based on TinyVit object location and graph convolution network

Fine-grained image classification is a branch of image classification. Recently, vision transformer has made excellent progress in the field of image recognition. Its self-attention mechanism can extract very effective image feature information. However, feeding fixed-size image blocks into the network introduces additional noise, which is detrimental to extract discriminative features for fine-grained images. The vision transformer's network model is large, making it difficult to utilize in practice. Moreover, many of today's fine-grained image classification methods focus on mining discriminative features while ignoring the connections within the image. To address these problems, we propose a novel method based on the lightweight TinyVit backbone network. Our approach utilizes the self-attention weight values of TinyVit as a guide to construct an effective object location (OL) module that cuts and enlarges the object area, providing the network with the opportunity to concentrate on the local object. Additionally, we employ the graph convolutional network (GCN) to create a spatial relationship feature learning (SRFL) module that captures spatial context information between image blocks in TinyVit with the help of the transformer's self-attention weights. OL and SRFL collaborate to jointly guide the classification task. The experimental results show that the proposed method achieved competitive performance, with the second-highest classification faccuracy on both the CUB-200–2011 and NABirds datasets. When tested on the Stanford Dogs dataset, our approach outperformed many popular methods. Our code is uploaded on https://github.com/hhhj1999/SRFL_OL.

Complete 3D Relationships Extraction Modality Alignment Network for 3D Dense Captioning.

3D dense captioning aims to semantically describe each object detected in a 3D scene, which plays a significant role in 3D scene understanding. Previous works lack a complete definition of 3D spatial relationships and the directly integrate visual and language modalities, thus ignoring the discrepancies between the two modalities. To address these issues, we propose a novel complete 3D relationship extraction modality alignment network, which consists of three steps: 3D object detection, complete 3D relationships extraction, and modality alignment caption. To comprehensively capture the 3D spatial relationship features, we define a complete set of 3D spatial relationships, including the local spatial relationship between objects and the global spatial relationship between each object and the entire scene. To this end, we propose a complete 3D relationships extraction module based on message passing and self-attention to mine multi-scale spatial relationship features and inspect the transformation to obtain features in different views. In addition, we propose the modality alignment caption module to fuse multi-scale relationship features and generate descriptions to bridge the semantic gap from the visual space to the language space with the prior information in the word embedding, and help generate improved descriptions for the 3D scene. Extensive experiments demonstrate that the proposed model outperforms the state-of-the-art methods on the ScanRefer and Nr3D datasets.

A Deep-Learning-Based Multimodal Data Fusion Framework for Urban Region Function Recognition

Accurate and efficient classification maps of urban functional zones (UFZs) are crucial to urban planning, management, and decision making. Due to the complex socioeconomic UFZ properties, it is increasingly challenging to identify urban functional zones by using remote-sensing images (RSIs) alone. Point-of-interest (POI) data and remote-sensing image data play important roles in UFZ extraction. However, many existing methods only use a single type of data or simply combine the two, failing to take full advantage of the complementary advantages between them. Therefore, we designed a deep-learning framework that integrates the above two types of data to identify urban functional areas. In the first part of the complementary feature-learning and fusion module, we use a convolutional neural network (CNN) to extract visual features and social features. Specifically, we extract visual features from RSI data, while POI data are converted into a distance heatmap tensor that is input into the CNN with gated attention mechanisms to extract social features. Then, we use a feature fusion module (FFM) with adaptive weights to fuse the two types of features. The second part is the spatial-relationship-modeling module. We designed a new spatial-relationship-learning network based on a vision transformer model with long- and short-distance attention, which can simultaneously learn the global and local spatial relationships of the urban functional zones. Finally, a feature aggregation module (FGM) utilizes the two spatial relationships efficiently. The experimental results show that the proposed model can fully extract visual features, social features, and spatial relationship features from RSIs and POIs for more accurate UFZ recognition.

Spatial state analysis of ship during berthing and unberthing process based on 3D LiDAR

In smart ports, accurate perception of vessels' state during berthing and unberthing is crucial to ensure the safety of vessels and ports. In this paper, a method for better monitoring the berthing and unberthing state of vessels is proposed by introducing the GIS spatial analysis technique. For the aspect of the motion state of the vessels, the optimal segmentation region of the hull from the point cloud data is extracted with the aid of GIS spatial theory. Subsequently, the key points are obtained from the optimal segmentation region to calculate the key parameters in the process of berthing and unberthing. Besides, the spatial raster analysis technique is employed to construct the missing point cloud data in the segmentation region. For the other aspect of the spatial state of the vessels, the spatial trajectory of vessels is given. According to the information on a spatial trajectory, a comprehensive study of the spatial state of vessels is performed from the perspective of directional features and spatial relationships. In summary, the proposed method not only identifies the vessels’ motion state from a spatial perspective but also accurately calculates key parameters related to berthing and unberthing. Furthermore, it provides intuitive and clear visualization of the spatial relationship features between the vessel and the shore. Based on the experimental data of berthing and unberthing in Dalian Port, China, accurate results are achieved to verify the feasibility of the proposed method, which can provide a new idea for intelligent berthing and unberthing of vessels.

Chest X-Ray Image Annotation based on Spatial Relationship Feature Extraction

Digital imaging has become an essential element in every medical institution. Therefore, medical image retrieval such as chest X-ray (CXR) must be improved via novel feature extraction and annotation activities before they are stored into image databases. To date, many methods have been introduced to annotate medical images using spatial relationships after these features are extracted. However, the annotation performance for each method is inconsistent and does not show promising achievement to retrieve images. It is noticed that each method is still struggling with at least two big problems. Firstly, the recommended annotation model is weak because the method does not consider the object shape and rely on gross object shape estimation. Secondly, the suggested annotation model can only be functional for simple object placement. As a result, it is difficult to determine the spatial relationship feature after they are extracted to annotate images accurately. Hence, this study aims to propose a new model to annotate nodule location within lung zone for CXR image with extracted spatial relationship feature to improve image retrieval. In order to achieve the aim, a methodology that consists of six phases of CXR image annotation using the extracted spatial relationship features is introduced. This comprehensive methodology covers all cycles for image annotation tasks starting from image pre-processing until determination of spatial relationship features for the lung zone in the CXR. The outcome from applying the methodology also enables us to produce a new semi-automatic annotation system named CHEXRIARS which acts as a tool to annotate the extracted spatial relationship features in CXR images. The CHEXRIARS performance is tested using a retrieval test with two common tests namely the precision and recall (PNR). Apart from CHEXRIARS, three other annotation methods that are object slope, object projection and comparison of region boundaries are also included in the retrieval performance test. Overall, the CHEXRIARS interpolated PNR curve shows the best shape because it is the closest curve approaching the value of 1 on the X-axis and Y-axis. Meanwhile the value of area under curve for CHEXRIARS also revealed that this system attained the highest score at 0.856 as compared to the other three annotation methods. The outcome from the retrieval performance test indicated that the proposed annotation model has produced outstanding outcome and improved the image retrieval.

Interactive Skin Wound Segmentation Based on Feature Augment Networks.

Skin wound segmentation in photographs allows non-invasive analysis of wounds that supports dermatological diagnosis and treatment. In this paper, we propose a novel feature augment network (FANet) to achieve automatic segmentation of skin wounds, and design an interactive feature augment network (IFANet) to provide interactive adjustment on the automatic segmentation results. The FANet contains the edge feature augment (EFA) module and the spatial relationship feature augment (SFA) module, which can make full use of the notable edge information and the spatial relationship information be-tween the wound and the skin. The IFANet, with FANet as the backbone, takes the user interactions and the initial result as inputs, and outputs the refined segmentation result. The pro-posed networks were tested on a dataset composed of miscellaneous skin wound images, and a public foot ulcer segmentation challenge dataset. The results indicate that the FANet gives good segmentation results while the IFANet can effectively improve them based on simple marking. Comprehensive comparative experiments show that our proposed networks outperform some other existing automatic or interactive segmentation methods, respectively.

CDSTAEP: Cross-Domain Spatial-Temporal Association Learning for Abnormal Events Prediction

Predicting network abnormal events and behavior can enhance security situation awareness and the ability to infer attack intentions. Most of the existing abnormal event prediction methods usually rely on the temporal relationship features between events and the spatial relationship features between hosts. However, the existing spatio-temporal anomaly event prediction methods do not fully consider the spatial relationship between events and the cross-domain environment of the behavior, resulting in poor performance in practical applications. In addition, the existing methods are mostly based on Euclidean space and hyperbolic space in terms of feature space relationship representation and do not fully consider the complexity of the relationship structure of anomalous events. In this paper, we propose a cross-domain spatio-temporal abnormal events prediction method, referred to as CDSTAEP. This method divides the local event sequence based on the temporal behavior sequence of entities and realizes the graphical representation of the multi-domain event correlation relationship. In the mixed-curvature space, we realize the representation learning of the correlation relationship of complex events and combine the event mixed-curvature vector representation and attention-based long short-term memory (LSTM-ATT) to capture the spatial and temporal correlation characteristics of cross-domain events, and finally realize the prediction. In this paper the proposed CDSTAEP is verified with the live network data set collected by a national key research and development plan. The results demonstrate that CDSTAEP can retain more spatial relationship features between events, the area under roc curve (AUC) score is better than the result of single-space representation and is 4.53% and 6.699% higher than the baseline models such as LSTM and LSTM-ATT.

Optimizing Spatial Relationships in GCN to Improve the Classification Accuracy of Remote Sensing Images

Semantic segmentation of remote sensing images is one of the core tasks of remote sensing image interpretation. With the continuous development of artificial intelligence technology, the use of deep learning methods for interpreting remote-sensing images has matured. Existing neural networks disregard the spatial relationship between two targets in remote sensing images. Semantic segmentation models that combine convolutional neural networks (CNNs) and graph convolutional neural networks (GCNs) cause a lack of feature boundaries, which leads to the unsatisfactory segmentation of various target feature boundaries. In this paper, we propose a new semantic segmentation model for remote sensing images (called DGCN hereinafter), which combines deep semantic segmentation networks (DSSN) and GCNs. In the GCN module, a loss function for boundary information is employed to optimize the learning of spatial relationship features between the target features and their relationships. A hierarchical fusion method is utilized for feature fusion and classification to optimize the spatial relationship information in the original feature information. Extensive experiments on ISPRS 2D and DeepGlobe semantic segmentation datasets show that compared with the existing semantic segmentation models of remote sensing images, the DGCN significantly optimizes the segmentation effect of feature boundaries, effectively reduces the noise in the segmentation results and improves the segmentation accuracy, which demonstrates the advancements of our model.

A novel pedal musculoskeletal response based on differential spatio-temporal LSTM for human activity recognition

Human activity recognition (HAR) with wearable devices has shown superior usability in everyday life tracking and healthcare monitoring in recent years. To solve the existing challenges of weak sensor durability and difficulty capturing dynamic features of traditional wearable devices based on the long short-term memory neural network (LSTM), a new HAR system based on a pedal wearable device is proposed by utilizing a novel differential spatio-temporal LSTM (DST-LSTM) method. The pedal wearable device is embedded in the tongue area of the general footwear to obtain pedal musculoskeletal response (PMR) data of dorsum pedis. Based on the PMR data, the DST-LSTM is developed to classify five typical activity statuses as standing, sitting, floor walking, down the stairs and up the stairs. First, the dynamic differential information of the PMR data is taken into consideration to build a new LSTM unit. Second, the spatial relationship features of the PMR data are obtained by multi-head graph attention networks (GAT). Third, a novel spatial gate is added to the new LSTM unit to filter irrelevant spatial features and get valid information. The effectiveness and superiority of both the pedal wearable device and the DST-LSTM method are validated by comparison with the state-of-the-art methods and wearable devices.

Distance Weight-Graph Attention Model-Based High-Resolution Remote Sensing Urban Functional Zone Identification

The spatial arrangement of land-cover features constitutes different urban functional zone. With the same attributes of the urban functional zone, the land-cover features that make up the functional zone will have similar spatial distribution characteristics. Considering the importance of understanding spatial relationships between land-cover features, the up-bottom hierarchical decomposition and semantic understanding of functional zone are achieved. First, for object convolution neural network (OCNN)-based land cover classification, an equal-area dividing algorithm is proposed to automatically generate convolution kernel position. Second, to extract spatial relationship features of urban land covers, a novel distance weight-graph attention model (DW-GAM) is originally proposed for classifying urban functional zones by comparing the feature similarity of the land cover relationship graph. Third, considering the extreme difficulties in expressing the urban structure characteristic on a single scale, a recursive model that uses an urban road network of different levels to divide multiscale functional zones is built. Finally, taking the analysis of urban function allocation as the application objective, this article establishes a primary framework of the spatial pattern evaluation index. Experimental results conducted on a Google Earth image of Xi'an city show that the multiscale recursive model can accurately recognize urban functional zones by using the originally proposed DW-GAM. Then, based on the outcome of urban functional zone identification, the case study of urban function allocation analysis is innovatively conducted on the fine scale to give some suggestions on future urban planning, which is, hence, of great significance for urban function pattern analysis and urban planning.

Pixel-Level Prediction for Ocean Remote Sensing Image Features Fusion Based on Global and Local Semantic Relations

With the rapid development of remote-sensing imaging technology, remote-sensing images have become increasingly diverse, and people are paying more attention to ocean remote-sensing research. Because ocean remote-sensing data are complex, and the ocean environment is diverse, results will differ, even if the same target is detected at different times in the same scene. To obtain more semantic features and better pixel-level prediction capabilities, this paper proposes a pixel-level ocean remote-sensing image algorithm (GLPO-Net) that combines local and global features. First, texture features, color features, and spatial relationship features are extracted. Second, the algorithm constructs a multiscale local cross-attention mechanism strategy to obtain feature weight information in different directions to fully mine the local features of ocean remote-sensing images. Concurrently, an algorithm constructs a multiscale global cross-attention mechanism strategy to obtain global features. Then, the fusion of global features and local features is described in each submodule to obtain more representative deep features. Finally, small-sample ocean remote-sensing is described via image pixel-level prediction. The algorithm proposed in this paper has been tested with three public ocean remote-sensing datasets. The experimental results show that the proposed GLPO-Net algorithm can learn features from small samples of ocean remote-sensing images. Compared to the prediction results of other remote-sensing image algorithms, GLPO-Net exhibits better prediction capabilities.

SRF‐Net: Spatial Relationship Feature Network for Tooth Point Cloud Classification

Abstract3D scanned point cloud data of teeth is popular used in digital orthodontics. The classification and semantic labelling for point cloud of each tooth is a key and challenging task for planning dental treatment. Utilizing the priori ordered position information of tooth arrangement, we propose an effective network for tooth model classification in this paper. The relative position and the adjacency similarity feature vectors are calculated for tooth 3D model, and combine the geometric feature into the fully connected layers of the classification training task. For the classification of dental anomalies, we present a dental anomalies processing method to improve the classification accuracy. We also use FocalLoss as the loss function to solve the sample imbalance of wisdom teeth. The extensive evaluations, ablation studies and comparisons demonstrate that the proposed network can classify tooth models accurately and automatically and outperforms state‐of‐the‐art point cloud classification methods.

A Multi-Type Features Method for Leg Detection in 2-D Laser Range Data

People detection is an important topic in the fields of security, intelligent environments, and robotics. Current research on people detection based on a single laser range finder is mostly focused on leg detection. However, in practical environments, where legs are likely to be touching or partially occluded, the current methods suffer from a low detection rate and precision. This paper proposes a multi-type features method for leg detection in 2-D laser range data. This method consists of segmentation, through which the laser range data are divided into segments; feature definition and extraction, in which three types of features, including relative distance statistical features, spatial relationship features and nearest neighbor features, are introduced and combined with classic geometric features; and classification, by which a strong classifier is generated using the real AdaBoost algorithm and segments are classified as leg or non-leg. Three 2-D laser range data sets are used for the experiments. The experimental results show that the proposed features are robust and effective in detecting both separated legs and touched or partially occluded legs.

The Application of an Anatomical Database for Fetal Congenital Heart Disease.

Background:Fetal congenital heart anomalies are the most common congenital anomalies in live births. Fetal echocardiography (FECG) is the only prenatal diagnostic approach used to detect fetal congenital heart disease (CHD). FECG is not widely used, and the antenatal diagnosis rate of CHD varies considerably. Thus, mastering the anatomical characteristics of different kinds of CHD is critical for ultrasound physicians to improve FECG technology. The aim of this study is to investigate the applications of a fetal CHD anatomic database in FECG teaching and training program.Methods:We evaluated 60 transverse section databases including 27 types of fetal CHD built in the Prenatal Diagnosis Center in Peking University People's Hospital. Each original database contained 400–700 cross-sectional digital images with a resolution of 3744 pixels × 5616 pixels. We imported the database into Amira 5.3.1 (Australia Visage Imaging Company, Australia) three-dimensional (3D) software. The database functions use a series of 3D software visual operations. The features of the fetal CHD anatomical database were analyzed to determine its applications in FECG continuing education and training.Results:The database was rebuilt using the 3D software. The original and rebuilt databases can be displayed dynamically, continuously, and synchronically and can be rotated at arbitrary angles. The sections from the dynamic displays and rotating angles are consistent with the sections in FECG. The database successfully reproduced the anatomic structures and spatial relationship features of different fetal CHDs. We established a fetal CHD anatomy training database and a standardized training database for FECG. Ultrasound physicians and students can learn the anatomical features of fetal CHD and FECG through either centralized training or distance education.Conclusions:The database of fetal CHD successfully reproduced the anatomic structures and spatial relationship of different kinds of fetal CHD. This database can be widely used in anatomy and FECG teaching and training.

Performance Comparison of Fourier Transform and its Derivatives as Shape Descriptors for Mango Grading

Mango is a tropical fruit of India which plays a major role in earning foreign currency by export. The export sector of India is paying attention towards it because of its commercial significance. Image has assorted inbuilt features which reflect its content such as color, texture, shape, and spatial relationship features, etc. How to organize and utilize these features effectively in agriculture era is a valuable research topic. Shape is a first quality factor to be considered by consumer while purchasing mango fruit. The purpose of this research work is to explore image processing algorithms and techniques to sort misshapen mango fruits based on their shape features. The developed algorithm would be first step in automated grading and sorting machines in export industries. It can provide a base for fully automatic grading system using computer vision. A shape based mango fruit sorting using computer vision is discussed in this paper. Shape features (Region based and contour based) are designed using Fourier Transform. Wavelet based descriptor is derived from basic Fourier transform to catch local shape details. A two layered radial basis neural network is developed to classify well formed and deformed mango fruits. The experiment results show performance comparison of all Fourier based shape descriptors. Wavelet Fourier Descriptor outperforms region based and contour based Fourier descriptor with classification efficiency of 89.83%.

Extracting Rural Settlement Information from Quickbird Images

Acquiring the information for rural settlement timely and accurately has an important significance for construction and development of rural areas. The development of remote sensing technology provides advanced means of the acquirement of the information of settlement. The study of extracting rural settlement information from Quickbird images in Xindu district, Chengdu City, P.R.of China was discussed here. Firstly, The Quickbird images such as panchromatic image and multi-spectral images were processed by geometric correction, enhancement and fusion. Secondly, the homogeneous image objects were formed by using multi-scale segmentation technology based on knowledge. Thirdly, the features such spectral feature, spatial relationship feature, texture feature and geometric feature of the image objects were obtained for each image object by using feature calculation. Fourthly, the feature knowledge of rural settlement unit and its component were obtained by using knowledge discovering. Finally, the rural settlement unit and its component information were extracted by matching the features with the feature knowledge of rural settlement unit and its component based on reasoning. It was shown that the rural settlement unit and its component information can be effectively extracted from Quickbird images by using our proposed method in this paper.

A Qualitative Approach to Content-Based Color Image Retrieval with Chinese Captions

This paper presents an intelligent method of retrieving images with Chinese captions from an image database. We combine color, shape and spatial features of the image to index and measure the similarity of images. As a technical contribution, a Seed-Filling like algorithm that could extract the shape and spatial relationship feature of images is proposed. Due to the difficulty of determining how far objects separate, we use qualitative spatial relations to analyze object similarities. Also, the system is incorporated with a visual interface and a set of tools, which allows the users to express the query by specifying or sketching the images conveniently. Besides, our feedback learning mechanism enhances the precision of retrieval. Our experience shows that the system is able to retrieve image information efficiently by the proposed approaches.