Fine-grained Information Research Articles

A Review-based Feature-level Information Aggregation Model for Graph Collaborative Filtering

To explicitly exploit the collaborative signals in the user-item interaction graph, a growing number of recent Collaborative Filtering (CF) studies adopt Graph Convolution Network (GCN) as a basis. Though effective, these methods basically treat all neighbors equally, ignoring the fact that neighbors should be target-related. While there are some works that assign different weights to neighbors via using the attention mechanism, they still suffer from two problems. First, the performance of them is limited by the lack of fine-grained details. Second, attention learned solely from interaction data may not reflect the user’s opinions accurately. To address these issues, we propose a Review-based Feature-level Information Aggregation (RFIA) graph model that incorporates the review information into the graph propagation to achieve more fine-grained information aggregation. The main idea of RFIA is to assign feature-level attention vectors for the interaction edges to adaptively adjust the contribution ratios of input neighbors across various dimensions, based on rich review information. Specifically, we first extract review features from text by BERT-Whitening. Then, we design non-linear feature extractors separately in two directions to further extract and refine these review features as feature-level attention. Finally, we design a graph contrastive learning module to optimize the learning of extractors under limited user behaviors. Experiments on three publicly available datasets validate the effectiveness and performance superiority of our proposed model.

Hybrid DAER Based Cross-Modal Retrieval Exploiting Deep Representation Learning.

Information retrieval across multiple modes has attracted much attention from academics and practitioners. One key challenge of cross-modal retrieval is to eliminate the heterogeneous gap between different patterns. Most of the existing methods tend to jointly construct a common subspace. However, very little attention has been given to the study of the importance of different fine-grained regions of various modalities. This lack of consideration significantly influences the utilization of the extracted information of multiple modalities. Therefore, this study proposes a novel text-image cross-modal retrieval approach that constructs a dual attention network and an enhanced relation network (DAER). More specifically, the dual attention network tends to precisely extract fine-grained weight information from text and images, while the enhanced relation network is used to expand the differences between different categories of data in order to improve the computational accuracy of similarity. The comprehensive experimental results on three widely-used major datasets (i.e., Wikipedia, Pascal Sentence, and XMediaNet) show that our proposed approach is effective and superior to existing cross-modal retrieval methods.

DPACFuse: Dual-Branch Progressive Learning for Infrared and Visible Image Fusion with Complementary Self-Attention and Convolution.

Infrared and visible image fusion aims to generate a single fused image that not only contains rich texture details and salient objects, but also facilitates downstream tasks. However, existing works mainly focus on learning different modality-specific or shared features, and ignore the importance of modeling cross-modality features. To address these challenges, we propose Dual-branch Progressive learning for infrared and visible image fusion with a complementary self-Attention and Convolution (DPACFuse) network. On the one hand, we propose Cross-Modality Feature Extraction (CMEF) to enhance information interaction and the extraction of common features across modalities. In addition, we introduce a high-frequency gradient convolution operation to extract fine-grained information and suppress high-frequency information loss. On the other hand, to alleviate the CNN issues of insufficient global information extraction and computation overheads of self-attention, we introduce the ACmix, which can fully extract local and global information in the source image with a smaller computational overhead than pure convolution or pure self-attention. Extensive experiments demonstrated that the fused images generated by DPACFuse not only contain rich texture information, but can also effectively highlight salient objects. Additionally, our method achieved approximately 3% improvement over the state-of-the-art methods in MI, Qabf, SF, and AG evaluation indicators. More importantly, our fused images enhanced object detection and semantic segmentation by approximately 10%, compared to using infrared and visible images separately.

TopicBubbler: An interactive visual analytics system for cross-level fine-grained exploration of social media data

How to explore fine-grained but meaningful information from the massive amount of social media data is critical but challenging. To address this challenge, we propose the TopicBubbler, a visual analytics system that supports the cross-level fine-grained exploration of social media data. To achieve the goal of cross-level fine-grained exploration, we propose a new workflow. Under the procedure of the workflow, we construct the fine-grained exploration view through the design of bubble-based word clouds. Each bubble contains two rings that can display information through different levels, and recommends six keywords computed by different algorithms. The view supports users collecting information at different levels and to perform fine-grained selection and exploration across different levels based on keyword recommendations. To enable the users to explore the temporal information and the hierarchical structure, we also construct the Temporal View and Hierarchical View, which satisfy users to view the cross-level dynamic trends and the overview hierarchical structure. In addition, we use the storyline metaphor to enable users to consolidate the fragmented information extracted across levels and topics and ultimately present it as a complete story. Case studies from real-world data confirm the capability of the TopicBubbler from different perspectives, including event mining across levels and topics, and fine-grained mining of specific topics to capture events hidden beneath the surface.

Exploiting deep transformer models in textual review based recommender systems

Textual reviews contain fine-grained information that can effectively infer user preferences over the items. Accordingly, the latest studies in the field of recommender systems exploit content-rich review texts to complement user and item representations and improve the ability to make personalized recommendations. Furthermore, the interactive deep learning mechanism can better model the user-item interaction from fine-grained textual reviews compared to traditional recommendation approaches improving the predictive performances. Therefore, it becomes important to investigate the design of existing deep learning methods for review-based recommender systems and innovate to make them capable of meeting desired recommendation schemes. The purpose of this research is to explore the performance of deep learning networks and deep transformer models in review-based recommender systems. In this paper, we conduct a compendious survey of the latest deep learning techniques in review-based recommender systems. Then investigation calls to employ and analyze deep transformer models for the review-based recommender systems. The wide range of experiments shows that deep transformer models can extract interpretable and relevant user/item representations than traditional deep learning networks. The findings indicate that the best deep transformer performance gains the maximum relative improvement (RMSE = 4.6%, MAE = 7.4%) with Amazon electronics, compared to the best outcome from traditional deep learning networks. In the end, this article highlights research gaps and outlines research opportunities for future research in this field.

TopoBERT: a plug and play toponym recognition module harnessing fine-tuned BERT

ABSTRACT Extracting precise geographical information from the textual content, referred to as toponym recognition, is fundamental in geographical information retrieval and crucial in a plethora of spatial analyses, e.g. mining location-based information from social media, news reports, and surveys for various applications. However, the performance of existing toponym recognition methods and tools is deficient in supporting tasks that rely on extracting fine-grained geographic information from texts, e.g. locating people sending help requests with addresses through social media during disasters. The emerging pretrained language models have revolutionized natural language processing and understanding by machines, offering a promising pathway to optimize toponym recognition to underpin practical applications. In this paper, TopoBERT, a uniquely designed toponym recognition module based on a one-dimensional Convolutional Neural Network (CNN1D) and Bidirectional Encoder Representation from Transformers (BERT), is proposed and fine-tuned. Three datasets are leveraged to tune the hyperparameters and discover the best strategy to train the model. Another seven datasets are used to evaluate the performance. TopoBERT achieves state-of-the-art performance (average f1-score = 0.854) compared to the seven baseline models. It is encapsulated into easy-to-use python scripts and can be seamlessly applied to diverse toponym recognition tasks without additional training.

National-Standards- and Deep-Learning-Oriented Raster and Vector Benchmark Dataset (RVBD) for Land-Use/Land-Cover Mapping in the Yangtze River Basin

A high-quality remote sensing interpretation dataset has become crucial for driving an intelligent model, i.e., deep learning (DL), to produce land-use/land-cover (LULC) products. The existing remote sensing datasets face the following issues: the current studies (1) lack object-oriented fine-grained information; (2) they cannot meet national standards; (3) they lack field surveys for labeling samples; and (4) they cannot serve for geographic engineering application directly. To address these gaps, the national-standards- and DL-oriented raster and vector benchmark dataset (RVBD) is the first to be established to map LULC for conducting soil water erosion assessment (SWEA). RVBD has the following significant innovation and contributions: (1) it is the first second-level object- and DL-oriented dataset with raster and vector data for LULC mapping; (2) its classification system conforms to the national industry standards of the Ministry of Water Resources of the People’s Republic of China; (3) it has high-quality LULC interpretation accuracy assisted by field surveys rather than indoor visual interpretation; and (4) it could be applied to serve for SWEA. Our dataset is constructed as follows: (1) spatio-temporal-spectrum information is utilized to perform automatic vectorization and label LULC attributes conforming to the national standards; and (2) several remarkable DL networks (DenseNet161, HorNet, EfficientNetB7, Vision Transformer, and Swin Transformer) are chosen as the baselines to train our dataset, and five evaluation metrics are chosen to perform quantitative evaluation. Experimental results verify the reliability and effectiveness of RVBD. Each chosen network achieves a minimum overall accuracy of 0.81 and a minimum Kappa of 0.80, and Vision Transformer achieves the best classification performance with overall accuracy of 0.87 and Kappa of 0.86. It indicates that RVBD is a significant benchmark, which could lay a foundation for intelligent interpretation of relevant geographic research about SWEA in the Yangtze River Basin and promote artificial intelligence technology to enrich geographical theories and methods.

Plural relational green space values for whom, when, and where? – A social media approach

The values people ascribe to their interactions with and within the environment are essential to inform justice and sustainability transformations. The development of many of these values unfolds through enjoying so-called cultural ecosystem services (CES) such as outdoor recreation, landscape aesthetics or environmental education. A growing body of literature is improving the assessment of the multiple ways that people value human and non-human relations arising when enjoying CES. Yet, the geo-temporal-demographic patterns of values distribution and the lessons that can be derived are to be consistently analysed within this relational framework. Building on a visual and textual content analysis of social media (SM) data geotagged in a peri-urban park of Barcelona, Spain, this research explores the potential of analysing the associated metadata (such as geotag, timestamp and social media users' demographics – i.e., performed gender and residency) in order to develop a better understanding of the linkages between people's values and the situated context of their construction. Our results show trends in relational CES values distribution along and between the analysed spatial, temporal, and demographic dimensions. In particular, despite there being a multiplicity of values revealed across the whole case-study area, to enjoy contemplative CES, such as spiritual or cognitive value, people need to move away from highly frequented areas and prefer specific times of the day, respectively evening or afternoon. Locals show a higher preference to visit the park on weekends compared to non-locals, while women-performing users show a significantly higher drop in their CES benefits uptake compared to men-performing users at night. In addition to providing novel and fine-grained information for transformative practices toward justice and sustainability, this study highlights the importance of complementing CES studies employing SM with metadata analysis to improve our understanding of the relationship between the real and the more-than-real.

Instance segmentation of individual tree crowns with YOLOv5: A comparison of approaches using the ForInstance benchmark LiDAR dataset

Fine-grained information on the level of individual trees constitute key components for forest observation enabling forest management practices tackling the effects of climate change and the loss of biodiversity in forest ecosystems. Such information on individual tree crowns (ITC's) can be derived from the application of ITC segmentation approaches, which utilize remotely sensed data. However, many ITC segmentation approaches require prior knowledge about forest characteristics, which is difficult to obtain for parameterization. This can be avoided by the adoption of data-driven, automated workflows based on convolutional neural networks (CNN). To contribute to the advancements of efficient ITC segmentation approaches, we present a novel ITC segmentation approach based on the YOLOv5 CNN. We analyzed the performance of this approach on a comprehensive international unmanned aerial laser scanning (UAV-LS) dataset (ForInstance), which covers a wide range of forest types. The ForInstance dataset consists of 4192 individually annotated trees in high-density point clouds with point densities ranging from 498 to 9529 points m-2 collected across 80 sites. The original dataset was split into 70% for training and validation and 30% for model performance assessment (test data). For the best performing model, we observed a F1-score of 0.74 for ITC segmentation and a tree detection rate (DET %) of 64% in the test data. This model outperformed an ITC segmentation approach, which requires prior knowledge about forest characteristics, by 41% and 33% for F1-score and DET %, respectively. Furthermore, we tested the effects of reduced point densities (498, 50 and 10 points per m-2) on ITC segmentation performance. The YOLO model exhibited promising F1-scores of 0.69 and 0.62 even at point densities of 50 and 10 points m-2, respectively, which were between 27% and 34% better than the ITC approach that requires prior knowledge.Furthermore, the areas of ITC segments resulting from the application of the best performing YOLO model were close to the reference areas (RMSE = 3.19 m-2), suggesting that the YOLO-derived ITC segments can be used to derive information on ITC level.

MLAGG-Net: Multi-level aggregation and global guidance network for pancreatic lesion segmentation in histopathological images

Pancreatic cancer is one of the most aggressive and lethal malignancies in the world. Automatic and accurate segmentation of lesion regions from histopathological images is essential for disease diagnosis and analysis. Nevertheless, it remains a challenging task due to the complicated pathological manifestations of the lesion, including the ambiguity of boundaries, the heterogeneity of textures, and the large variations in morphology, size and location. To address these problems, we propose a multi-level aggregation and global guidance network. Specifically, we utilize a multi-level feature fusion and distribution module that aims to alleviate the scale diversity and complex boundary problems in lesions by aggregating fine-grained information of low-level features and semantic information of high-level features. The cross-attention fusion module is proposed to perform the aggregation of hierarchical multi-level features, which can absorb effective information and suppress redundant information from them. This paper also introduces a global information fusion module to guide the network to locate the lesion areas with inconsistent distribution between the internal and boundary regions more accurately for more complete segmentation results. We conduct experiments on our PANC dataset and the public GlaS challenge dataset to verify the effectiveness of the proposed network. The experimental results show that the proposed method can achieve better segmentation performance with 90.02% Dice and 82.07% Jaccard compared to other state-of-the-art methods on our PANC dataset while achieving competitive results on the GlaS challenge dataset.

Key Agreement on IoT Devices With Echo Profiling

Secure Device-to-Device (D2D) communication is important for the Internet-of-Things (IoT) devices. Key agreement between devices is the important first step in building a secure D2D channel. Due to the lack of third-party certification, key agreement for IoT devices has to rely on the untrusted channel between them, which makes the pairing process vulnerable to attacks such as eavesdropping, jamming, and predictable channel attack. We solve this problem with a novel key agreement method named, where two nearby devices can generate a symmetric key independently, leveraging the ambient sound signal that can be locally collected by devices. The intuition of this idea is that the propagation delays of the ambient sounds and the echoes will carry fine-grained information about the spatial context of the receiving device, which can be transformed to a special key for that device. So, nearby devices which have similar spatial context will generate similar echo profiles. We implement a prototype of and evaluate its performance in resisting different attacks. The results tell that, with, the key agreement process is even undetectable by an attacker which is only 50cm away from the pairing devices, and is thus resistant to all attacks mentioned above.

Pseudo-Mono for Monocular 3D Object Detection in Autonomous Driving

Current monocular 3D object detection algorithms generally suffer from inaccurate depth estimation, which leads to reduction of detection accuracy. The depth error from image-to-image generation for the stereo view is insignificant compared with the gap in single-image generation. Therefore, a novel pseudo-monocular 3D object detection framework is proposed, which is called Pseudo-Mono. Particularly, stereo images are brought into monocular 3D detection. Firstly, stereo images are taken as input, then a lightweight depth predictor is used to generate the depth map of input images. Secondly, the left input images obtained from stereo camera are used as subjects, which generate enhanced visual feature and multi-scale depth feature by depth indexing and feature matching probabilities, respectively. Finally, sparse anchors set by the foreground probability maps and the multi-scale feature maps are used as reference points to find the suitable initialization approach of object query. The encoded visual feature is adopted to enhance object query for enabling deep interaction between visual feature and depth feature. Compared with popular monocular 3D object detection methods, Pseudo-Mono is able to achieve richer fine-grained information without additional data input. Extensive experimental results on the datasets of KITTI, NuScenes, and MS-COCO demonstrate the generalizability and portability of the proposed method. The effectiveness and efficiency of Pseudo-Mono have been demonstrated by extensive ablation experiments. Experiments on a real vehicle platform have shown that the proposed method maintains high performance in complex real-world environments.

Multi-Task Learning U-Net for Functional Shoulder Sub-Task Segmentation.

The assessment of a frozen shoulder (FS) is critical for evaluating outcomes and medical treatment. Analysis of functional shoulder sub-tasks provides more crucial information, but current manual labeling methods are time-consuming and prone to errors. To address this challenge, we propose a deep multi-task learning (MTL) U-Net to provide an automatic and reliable functional shoulder sub-task segmentation (STS) tool for clinical evaluation in FS. The proposed approach contains the main task of STS and the auxiliary task of transition point detection (TPD). For the main STS task, a U-Net architecture including an encoder-decoder with skip connection is presented to perform shoulder sub-task classification for each time point. The auxiliary TPD task uses lightweight convolutional neural networks architecture to detect the boundary between shoulder sub-tasks. A shared structure is implemented between two tasks and their objective functions of them are optimized jointly. The fine-grained transition-related information from the auxiliary TPD task is expected to help the main STS task better detect boundaries between functional shoulder sub-tasks. We conduct the experiments using wearable inertial measurement units to record 815 shoulder task sequences collected from 20 healthy subjects and 43 patients with FS. The experimental results present that the deep MTL U-Net can achieve superior performance compared to using single-task models. It shows the effectiveness of the proposed method for functional shoulder STS. The code has been made publicly available at https://github.com/RobinChu9890/MTL-U-Net-for-Functional-Shoulder-STS.Clinical Relevance- This work provides an automatic and reliable functional shoulder sub-task segmentation tool for clinical evaluation in frozen shoulder.

Aircraft Detection and Fine-Grained Recognition Based on High-Resolution Remote Sensing Images

In order to realize the detection and recognition of specific types of an aircraft in remote sensing images, this paper proposes an algorithm called Fine-grained S2ANet (FS2ANet) based on the improved Single-shot Alignment Network (S2ANet) for remote sensing aircraft object detection and fine-grained recognition. Firstly, to address the imbalanced number of instances of various aircrafts in the dataset, we perform data augmentation on some remote sensing images using flip and color space transformation methods. Secondly, this paper selects ResNet101 as the backbone, combines space-to-depth (SPD) to improve the FPN structure, constructs the FPN-SPD module, and builds the aircraft fine feature focusing module (AF3M) in the detection head of the network, which reduces the loss of fine-grained information in the process of feature extraction, enhances the extraction capability of the network for fine aircraft features, and improves the detection accuracy of remote sensing micro aircraft objects. Finally, we use the SkewIoU based on Kalman filtering (KFIoU) as the algorithm’s regression loss function, improving the algorithm’s convergence speed and the object boxes’ regression accuracy. The experimental results of the detection and fine-grained recognition of 11 types of remote sensing aircraft objects such as Boeing 737, A321, and C919 using the FS2ANet algorithm show that the mAP0.5 of FS2ANet is 46.82%, which is 3.87% higher than S2ANet, and it can apply to the field of remote sensing aircraft object detection and fine-grained recognition.

Gaze Estimation Method Combining Facial Feature Extractor with Pyramid Squeeze Attention Mechanism

To address the issue of reduced gaze estimation accuracy caused by individual differences in different environments, this study proposes a novel gaze estimation algorithm based on attention mechanisms. Firstly, by constructing a facial feature extractor (FFE), the method obtains facial feature information about the eyes and locates the feature areas of the left and right eyes. Then, the L2CSNet (l2 loss + cross-entropy loss + softmax layer network), which integrates the PSA (pyramid squeeze attention), is designed to increase the correlation weights related to gaze estimation in the feature areas, suppress other irrelevant weights, and extract more fine-grained feature information to obtain gaze direction features. Finally, by integrating L2CSNet with FFE and PSA, FPSA_L2CSNet was proposed, which is fully tested on four representative publicly available datasets and a real-world dataset comprising individuals of different backgrounds, lighting conditions, nationalities, skin tones, ages, genders, and partial occlusions. The experimental results indicate that the accuracy of the gaze estimation model proposed in this paper has been improved by 13.88%, 11.43%, and 7.34%, compared with L2CSNet, FSE_L2CSNet, and FCBA_L2CSNet, respectively. This model not only improves the robustness of gaze estimation but also provides more accurate estimation results than the original model.

Unmanned Aerial Vehicle Perspective Small Target Recognition Algorithm Based on Improved YOLOv5

Small target detection has been widely used in applications that are relevant to everyday life and have many real-time requirements, such as road patrols and security surveillance. Although object detection methods based on deep learning have achieved great success in recent years, they are not effective in small target detection. In order to solve the problem of low recognition rate caused by factors such as low resolution of UAV viewpoint images and little valid information, this paper proposes an improved algorithm based on the YOLOv5s model, called YOLOv5s-pp. First, to better suppress interference from complex backgrounds and negative samples in images, we add a CA attention module, which can better focus on task-specific important channels while weakening the influence of irrelevant channels. Secondly, we improve the forward propagation and generalisation of the network using the Meta-ACON activation function, which adaptively learns to adjust the degree of linearity or nonlinearity of the activation function based on the input data. Again, the SPD Conv module is incorporated into the network model to address the problems of reduced learning efficiency and loss of fine-grained information due to cross-layer convolution in the model. Finally, the detection head is improved by using smaller, smaller-target detection heads to reduce missed detections. We evaluated the algorithm on the VisDrone2019-DET and UAVDT datasets and compared it with other state-of-the-art algorithms. Compared to YOLOv5s, mAP@.5 improved by 7.4% and 6.5% on the VisDrone2019-DET and UAVDT datasets, respectively, and compared to YOLOv8s, mAP@.5 improved by 0.8% and 2.1%, respectively. For improving the performance of the UAV-side small target detection algorithm, it will help to enhance the reliability and safety of UAVs in critical missions such as military reconnaissance, road patrol and security surveillance.

Quantification and Pictorial Expression of Driving Status Domain Boundaries for Autonomous Vehicles in LTAP/OD Scenarios

The ability of advanced driver assistance systems (ADAS) and autonomous vehicles to make human-like decisions can be enhanced by providing more detailed information about vehicles and in-vehicle users’ states. In this paper, the driving status domains of vehicles in left turn across path/opposite direction (LTAP/OD) scenarios are subdivided into comfort, discomfort, extreme, and crash, and the boundaries of each status domain are quantified and visualized. First, real unprotected left turn road segments are chosen for the actual vehicle testing. Subjective passenger comfort evaluation results and objective motion state data of vehicles during the experiment are organized and analyzed by statistics. In addition, the pictorials are plotted to determine the comfort and extreme status domain boundaries based on motion state parameters. Second, based on the unprotected left turn kinematic analysis and modeling, as well as a skilled driver risk perception and operational model, the Safe Collision Plots (SCP) of conflicting vehicles in LTAP/OD scenarios are quantified and expressed as pictorial examples. By combining objective motion parameters and passenger experience, intuitively quantifying each driving status domain of vehicles can provide more fine-grained information for the design parameters of ADAS and autonomous vehicles and increase public trust and acceptance of them.

SP-YOLOv8s: An Improved YOLOv8s Model for Remote Sensing Image Tiny Object Detection

An improved YOLOv8s-based method is proposed to address the challenge of accurately recognizing tiny objects in remote sensing images during practical human-computer interaction. In detecting tiny targets, the accuracy of YOLOv8s is low because the downsampling module of the original YOLOv8s algorithm causes the network to lose fine-grained feature information, and the neck network feature information needs to be sufficiently fused. In this method, the strided convolution module in YOLOv8s is replaced with the SPD-Conv module. By doing so, the feature map undergoes downsampling while preserving fine-grained feature information, thereby improving the learning and expressive capabilities of the network and enhancing recognition accuracy. Meanwhile, the path aggregation network is substituted with the SPANet structure, which facilitates the acquisition of more prosperous gradient paths. This substitution enhances the fusion of feature maps at various scales, reduces model parameters, and further improves detection accuracy. Additionally, it enhances the network’s robustness to complex backgrounds. Experimental verification is conducted on the following two intricate datasets containing tiny objects: AI-TOD and TinyPerson. A comparative analysis with the original YOLOv8s algorithm reveals notable enhancements in recognition accuracy. Specifically, under real-time performance constraints, the proposed method yields a 4.9% and 9.1% improvement in mAP0.5 recognition accuracy for AI-TOD and TinyPerson datasets, respectively. Moreover, the recognition accuracy for mAP0.5:0.95 is enhanced by 3.4% and 3.2% for the same datasets, respectively. The results indicate that the proposed method enables rapid and accurate recognition of tiny objects in complex backgrounds. Furthermore, it demonstrates better recognition precision and stability than other algorithms, such as YOLOv5s and YOLOv8s.

HyperGraph based human mesh hierarchical representation and reconstruction from a single image

Reconstructing 3D human mesh from monocular images has been extensively studied. However, the existing non-parametric reconstruction methods are inefficient when modeling vertex relationship concerning human information due to they generally adopt an uniform template mesh. To this end, this paper proposes a novel hypergraph-based human mesh hierarchical representation that enables the expression of vertices at body, parts, and vertices perspectives, corresponding to global, local and individual, respectively. And then a novel HyperGraph Attention-based human mesh reconstruction network (HGaMRNet) is put forward in turn, which mainly consists of two modules and can efficiently capture human information from different granularities of the mesh. Specifically, the first module, Body2Parts, decouples a body into local parts, and leverages Mix-Attention (MAT) based feature encoder to learn visual cues and semantic information of the parts for capturing complex human kinematic relationships from monocular images. The second module, Part2Vertices, transfers part features to the corresponding vertices through an adaptive incidence matrix, and utilizes a HyperGraph Attention network to update the vertex features. This is conductive to learning the fine-grained morphological information of a human mesh. All in one, supported by the hypergraph-based hierarchical representation of the human mesh, HGaMRNet balances the effects of neighbor vertex from different levels properly and eventually promotes the reconstruction accuracy of human mesh. Experiments conducted on both Human3.6M and 3DPW datasets show that HGaMRNet outperforms most of the existing image-based human mesh reconstruction methods.

Multi-level attention for referring expression comprehension

Referring expression comprehension aims to locate a target object in an image described by a referring expression, where extracting semantic and discriminative visual information plays an important role. Most existing methods either ignore attribute information or context information in the model learning procedure, thus resulting in less effective visual features. In this paper, we propose a Multi-level Attention Network (MANet) to extract the target attribute information and the surrounding context information simultaneously for the target object, where the Attribute Attention Module is designed to extract the fine-grained visual information related to the referring expression and the Context Attention Module is designed to merge the context information of surroundings to learn more discriminative visual features. Experiments on various common benchmark datasets show the effectiveness of our approach.