High-level Semantic Information Research Articles

Salient Object Detection Combining a Self-Attention Module and a Feature Pyramid Network

Salient object detection has achieved great improvements by using the Fully Convolutional Networks (FCNs). However, the FCN-based U-shape architecture may cause dilution problems in the high-level semantic information during the up-sample operations in the top-down pathway. Thus, it can weaken the ability of salient object localization and produce degraded boundaries. To this end, in order to overcome this limitation, we propose a novel pyramid self-attention module (PSAM) and the adoption of an independent feature-complementing strategy. In PSAM, self-attention layers are equipped after multi-scale pyramid features to capture richer high-level features and bring larger receptive fields to the model. In addition, a channel-wise attention module is also employed to reduce the redundant features of the FPN and provide refined results. Experimental analysis demonstrates that the proposed PSAM effectively contributes to the whole model so that it outperforms state-of-the-art results over five challenging datasets. Finally, quantitative results show that PSAM generates accurate predictions and integral salient maps, which can provide further help to other computer vision tasks, such as object detection and semantic segmentation.

An End-to-End Framework for Clothing Collocation Based on Semantic Feature Fusion

In this article, we develop an end-to-end clothing collocation learning framework based on a bidirectional long short-term memories (Bi-LTSM) model, and propose new feature extraction and fusion modules. The feature extraction module uses Inception V3 to extract low-level feature information and the segmentation branches of Mask Region Convolutional Neural Network (RCNN) to extract high-level semantic information; whereas the feature fusion module creates a new reference vector for each image to fuse the two types of image feature information. As a result, the feature can involve both low-level image and high-level semantic feature information, so that the performance of Bi-LSTM can be enhanced. Extensive simulations are conducted based on Ployvore and DeepFashion2 datasets. Simulation results verify the effectiveness of the proposed method compared with other state-of-the-art clothing collocation methods.

Enhancing feature fusion for human pose estimation

Current human pose estimation methods mainly rely on designing efficient Convolutional Neural Networks (CNN) frameworks. These CNN architectures typically consist of high-to-low resolution sub-networks to learn semantic information, and then followed by low-to-high sub-networks to raise the resolution to locate the keypoints. Because low-level features have high resolution but less semantic information, while high-level features have rich semantic information but less high resolution details, so it is important to fuse different level features to improve the final performance. However, most existing models implement feature fusion by simply concatenate low-level and high-level features without considering the gap between spatial resolution and semantic levels. In this paper, we propose a new feature fusion method for human pose estimation. We introduce high level semantic information into low-level features to enhance feature fusion. Further, to keep both the high-level semantic information and high-resolution location details, we use Global Convolutional Network blocks to bridge the gap between low-level and high-level features. Experiments on MPII and LSP human pose estimation datasets demonstrate that efficient feature fusion can significantly improve the performance. The code is available at: https://github.com/tongjiangwei/FeatureFusion .

Multiclass Object Detection in UAV Images Based on Rotation Region Network

The object detection in UAV application is a challenging task due to the diversity of target scales, variation of views, and complex backgrounds. To solve several challenges, including dense objects, objects with arbitrary orientation, and diversity of aspect ratios, this article proposes an end-to-end object detection method based on the convolutional neural network. In this article, the feature extraction performance is enhanced by utilizing a deep residual neural network. Multiscale feature maps are obtained through fusion with different convolutional layers, thus combining the high-level semantic information and low-level detail information. A rotation region proposal network is adopted to generate rotated regions, which makes the bounding box sensitive to dense objects in aerial images. Meanwhile, the RoIAlign is used and a convolution layer is appended in the classification stage, and focal loss is used in the classification stage. The proposed method focuses on arbitrary-oriented and dense objects in UAV images. After a comprehensive evaluation with several state-of-the-art object detection algorithms, the proposed method is proved to be effective to detect multiclass artificial objects in aerial images. Extensive experiments are conducted on the DOTA, VEDAI, and the VisDrone UAV image datasets, which demonstrate that the proposed method can obtain discriminative features through the improved multiscale feature extraction and the rotating region network. The results on the above datasets show that our method obtains gains in mean average precision compared with several state-of-the-art methods.

Semantic preview benefit and cost: Evidence from parafoveal fast-priming paradigm

How is semantic information in the mental lexicon accessed and selected during reading? Readers process information of both the foveal and parafoveal words. Recent eye-tracking studies hint at bi-phasic lexical activation dynamics, demonstrating that semantically related parafoveal previews can either facilitate, or interfere with lexical processing of target words in comparison to unrelated previews, with the size and direction of the effect depending on exposure time to parafoveal previews. However, evidence to date is only correlational, because exposure time was determined by participants' pre-target fixation durations. Here we experimentally controlled parafoveal preview exposure duration using a combination of the gaze-contingent fast-priming and boundary paradigms. We manipulated preview duration and examined the time course of parafoveal semantic activation during the oral reading of Chinese sentences in three experiments. Semantic previews led to faster lexical access of target words than unrelated previews only when the previews were presented briefly (80 ms in Experiments 1 and 3). Longer exposure time (100 ms or 150 ms) eliminated semantic preview effects, and full preview without duration limit resulted in preview cost, i.e., a reversal of preview benefit. Our results indicate that high-level semantic information can be obtained from parafoveal words and the size and direction of the parafoveal semantic effect depends on the level of lexical activation.

FASSD: A Feature Fusion and Spatial Attention-Based Single Shot Detector for Small Object Detection

Deep learning methods have significantly improved object detection performance, but small object detection remains an extremely difficult and challenging task in computer vision. We propose a feature fusion and spatial attention-based single shot detector (FASSD) for small object detection. We fuse high-level semantic information into shallow layers to generate discriminative feature representations for small objects. To adaptively enhance the expression of small object areas and suppress the feature response of background regions, the spatial attention block learns a self-attention mask to enhance the original feature maps. We also establish a small object dataset (LAKE-BOAT) of a scene with a boat on a lake and tested our algorithm to evaluate its performance. The results show that our FASSD achieves 79.3% mAP (mean average precision) on the PASCAL VOC2007 test with input 300 × 300, which outperforms the original single shot multibox detector (SSD) by 1.6 points, as well as most improved algorithms based on SSD. The corresponding detection speed was 45.3 FPS (frame per second) on the VOC2007 test using a single NVIDIA TITAN RTX GPU. The test results of a simplified FASSD on the LAKE-BOAT dataset indicate that our model achieved an improvement of 3.5% mAP on the baseline network while maintaining a real-time detection speed (64.4 FPS).

Boundary-aware High-resolution Network with region enhancement for salient object detection

Existing deep learning based salient object detection methods have achieved gratifying progress, however they still suffer from the coarse boundaries and incompleteness of salient objects. To alleviate these issues, this paper presents a novel method that supplements the low-resolution high-level semantic information to high-resolution low-level information with boundary optimization and region enhancement. Based on the parallel architecture, we design a Boundary-aware High-resolution Network (BHNet). First, BHNet maintains high resolution to extract features of the image at the first pathway, and the resolution of the other four pathways are lower, which are used to provide more semantic information for the first pathway. Second, in order to better integrate multi-level semantic information into the first pathway and improve the ability of the model to perceive salient objects, several Multi-path Channel Weight Modules (MCWMs) and Region Enhancement Modules (REMs) are further designed for corresponding blocks. Finally, we also propose a boundary loss function to guide the network to learn more detailed boundary information, which leads to accurate predictions with clear boundaries. Exhaustive evaluations on 6 popular datasets illustrate that the proposed method outperforms the state-of-the-art approaches due to its superior performance, nice generalization and powerful learning ability.

OPD-Net: Prow Detection Based on Feature Enhancement and Improved Regression Model in Optical Remote Sensing Imagery

Accurate prow detection (i.e., ship heading prediction) is important in many applications that rely on optical remote sensing imagery, such as track forecasting and maritime navigation. In recent years, many advanced methods based on deep convolution neural networks (DCNNs) have succeeded in detecting multidirectional ships. However, these methods are not effective at determining the prow orientation, primarily due to three limitations: weak adaptability to geometric transformations of ship targets, confusing the semantic information between the prow and other parts of ships, and the boundary discontinuity problem. To address these problems, we propose an omnidirectional prow detection network (OPD-Net) based on feature enhancement and an improved regression model. OPD-Net consists of a feature refinement network (FRN), a prow attention network (PAN), and a complex plane coordinates regression model (CPCRM). First, the FRN balances the low-level location information and high-level semantic information from multiscale feature maps and then fits various geometric transformations regarding ship targets through deformable blocks. Next, the PAN, which is based on supervised learning, is used to enhance the ship prow feature as well as suppress background noise, which improves the accuracy of ship heading predictions. Finally, the CPCRM is designed to effectively solve the boundary discontinuity problem and correctly achieve prow detection in arbitrary orientations. Experiments on optical remote sensing image data sets demonstrate the robustness and superiority of our method for prow detection. Moreover, our approach is also competitive when used only for ship detection.

Medical image retrieval for detecting pneumonia using binary classification with deep convolutional neural networks

In the concepts of Deep Learning, the ConvNets are a classified group of deep neural networks that are used to analyze the images. They find their major applications in image classification and in the medical image analysis. ConvNets are regularized, multilayer preceptors. The major challenge today is the increasing size of medical image repositories which is leading to troubles in large database management and then querying those databases for storage and retrieval and this can only be sorted out using the Content Based Medical Image Retrieval (CBMIR) systems. A major challenge in CBMIR systems is the semantic gap that exists between the low-level visual information captured by the imaging devices and high-level semantic information captured by human. This paper demonstrates a new and efficient framework with the algorithm of Deep CNN for the Feature Based Medical Image Retrieval (FBMIR) system for fast and efficient retrieval of medical and clinical images for detecting Pneumonia. The intermodal datasets are used and divided into two classes to train the network. One class consists of the lungs infected by Pneumonia and the other class contains the images of lungs which are normal. The learned features from the trained deep convolutional neural network with the fast processing feature of the hash method are used to reduce the feature space. To achieve the best retrieval results with class-based predictions 93.9% of average classification accuracy is achieved and 0.87 of mean average precision is achieved for this retrieval task. The proposed framework fit the best for top-ranked multimodal medical image retrieval.

Multi-level feature aggregation network for instrument identification of endoscopic images

Identification of surgical instruments is crucial in understanding surgical scenarios and providing an assistive process in endoscopic image-guided surgery. This study proposes a novel multilevel feature-aggregated deep convolutional neural network (MLFA-Net) for identifying surgical instruments in endoscopic images. First, a global feature augmentation layer is created on the top layer of the backbone to improve the localization ability of object identification by boosting the high-level semantic information to the feature flow network. Second, a modified interaction path of cross-channel features is proposed to increase the nonlinear combination of features in the same level and improve the efficiency of information propagation. Third, a multiview fusion branch of features is built to aggregate the location-sensitive information of the same level in different views, increase the information diversity of features, and enhance the localization ability of objects. By utilizing the latent information, the proposed network of multilevel feature aggregation can accomplish multitask instrument identification with a single network. Three tasks are handled by the proposed network, including object detection, which classifies the type of instrument and locates its border; mask segmentation, which detects the instrument shape; and pose estimation, which detects the keypoint of instrument parts. The experiments are performed on laparoscopic images from MICCAI 2017 Endoscopic Vision Challenge, and the mean average precision (AP) and average recall (AR) are utilized to quantify the segmentation and pose estimation results. For the bounding box regression, the AP and AR are 79.1% and 63.2%, respectively, while the AP and AR of mask segmentation are 78.1% and 62.1%, and the AP and AR of the pose estimation achieve 67.1% and 55.7%, respectively. The experiments demonstrate that our method efficiently improves the recognition accuracy of the instrument in endoscopic images, and outperforms the other state-of-the-art methods.

Prior Guided Feature Enrichment Network for Few-Shot Segmentation.

State-of-the-art semantic segmentation methods require sufficient labeled data to achieve good results and hardly work on unseen classes without fine-tuning. Few-shot segmentation is thus proposed to tackle this problem by learning a model that quickly adapts to new classes with a few labeled support samples. Theses frameworks still face the challenge of generalization ability reduction on unseen classes due to inappropriate use of high-level semantic information of training classes and spatial inconsistency between query and support targets. To alleviate these issues, we propose the Prior Guided Feature Enrichment Network (PFENet). It consists of novel designs of (1) a training-free prior mask generation method that not only retains generalization power but also improves model performance and (2) Feature Enrichment Module (FEM) that overcomes spatial inconsistency by adaptively enriching query features with support features and prior masks. Extensive experiments on PASCAL-5 i and COCO prove that the proposed prior generation method and FEM both improve the baseline method significantly. Our PFENet also outperforms state-of-the-art methods by a large margin without efficiency loss. It is surprising that our model even generalizes to cases without labeled support samples.

ADR-Net: Context extraction network based on M-Net for medical image segmentation.

Medical image segmentation is an essential component of medical image analysis. Accurate segmentation can assist doctors in diagnosis and relieve their fatigue. Although several image segmentation methods based on U-Net have been proposed, their performances have been observed to be suboptimal in the case of small-sized objects. To address this shortcoming, a novel network architecture is proposed in this study to enhance segmentation performance on small medical targets. In this paper, we propose a joint multi-scale context attention network architecture to simultaneously capture higher level semantic information and spatial information. In order to obtain a greater number of feature maps during decoding, the network concatenates the images of side inputs by down-sampling during the encoding phase. In the bottleneck layer of the network, dense atrous convolution (DAC) and multi-scale residual pyramid pooling (RMP) modules are exploited to better capture high-level semantic information and spatial information. To improve the segmentation performance on small targets, the attention gate (AG) block is used to effectively suppress feature activation in uncorrelated regions and highlight the target area. The proposed model is first evaluated on the public dataset DRIVE, on which it performs significantly better than the basic framework in terms of sensitivity (SE), intersection-over-union (IOU), and area under the receiver operating characteristic curve (AUC). In particular, the SE and IOU are observed to increase by 7.46% and 5.97%, respectively. Further, the evaluation indices exhibit improvements compared to those of state-of-the-art methods as well, with SE and IOU increasing by 3.58% and 3.26%, respectively. Additionally, in order to demonstrate the generalizability of the proposed architecture, we evaluate our model on three other challenging datasets. The respective performances are observed to be better than those of state-of-the-art network architectures on the same datasets. Moreover, we use lung segmentation as a comparative experiment to demonstrate the transferability of the advantageous properties of the proposed approach in the context of small target segmentation to the segmentation of large targets. Finally, an ablation study is conducted to investigate the individual contributions of the AG block, the DAC block, and the RMP block to the performance of the network. The proposed method is evaluated on various datasets. Experimental results demonstrate that the proposed model performs better than state-of-the-art methods in medical image segmentation of small targets.

A Distributed Automatic Video Annotation Platform

In the era of digital devices and the Internet, thousands of videos are taken and share through the Internet. Similarly, CCTV cameras in the digital city produce a large amount of video data that carry essential information. To handle the increased video data and generate knowledge, there is an increasing demand for distributed video annotation. Therefore, in this paper, we propose a novel distributed video annotation platform that explores the spatial information and temporal information. Afterward, we provide higher-level semantic information. The proposed framework is divided into two parts: spatial annotation and spatiotemporal annotation. Therefore, we propose a spatiotemporal descriptor, namely, volume local directional ternary pattern-three orthogonal planes (VLDTP–TOP) in a distributed manner using Spark. Moreover, we developed several state-of-the-art appearance-based and spatiotemporal-based feature descriptors on top of Spark. We also provide the distributed video annotation services for the end-users so that they can easily use the video annotation and APIs for development to produce new video annotation algorithms. Due to the lack of a spatiotemporal video annotation dataset that provides ground truth for both spatial and temporal information, we introduce a video annotation dataset, namely, STAD which provides ground truth for spatial and temporal information. An extensive experimental analysis was performed in order to validate the performance and scalability of the proposed feature descriptors, which proved the excellence of our proposed approach.

A Journey from basic Image Features to Lofty Human Intelligence in Content-based Image Retrieval: Motivation, Applications and Future Trends

Due to a remarkable increase in the complexity of the multimedia content, there is a cumulative enhancement of digital images both online and offline. For the purpose of retrieving images from a vast storehouse of images, there is an urgent requirement of an effectual image retrieval system and the most effective system in this domain is denoted as content-based image retrieval (CBIR) system. CBIR system is generally based on the extraction of basic image attributes like texture, color, shape, spatial information, etc. from an image. But, there exists a semantic gap between the basic image features and high-level human perception and to reduce this gap various techniques can be used. This paper presents a detailed study about the various basic techniques with an emphasis on different intelligent techniques like, the usage of machine learning, deep learning, relevance feedback, etc., which can be used to achieve a high level semantic information in CBIR systems. In addition, a detailed outline regarding the framework of a basic CBIR system, various benchmark datasets, similarity measures, evaluation metrics have been also discussed. Finally, solution to some research issues and future trends have also been given in this paper.

A Novel Multi-Focus Image Fusion Network with U-Shape Structure.

Multi-focus image fusion has become a very practical image processing task. It uses multiple images focused on various depth planes to create an all-in-focus image. Although extensive studies have been produced, the performance of existing methods is still limited by the inaccurate detection of the focus regions for fusion. Therefore, in this paper, we proposed a novel U-shape network which can generate an accurate decision map for the multi-focus image fusion. The Siamese encoder of our U-shape network can preserve the low-level cues with rich spatial details and high-level semantic information from the source images separately. Moreover, we introduce the ResBlocks to expand the receptive field, which can enhance the ability of our network to distinguish between focus and defocus regions. Moreover, in the bridge stage between the encoder and decoder, the spatial pyramid pooling is adopted as a global perception fusion module to capture sufficient context information for the learning of the decision map. Finally, we use a hybrid loss that combines the binary cross-entropy loss and the structural similarity loss for supervision. Extensive experiments have demonstrated that the proposed method can achieve the state-of-the-art performance.

Automatic segmentation of intracerebral hemorrhage in CT images using encoder–decoder convolutional neural network

Intracerebral hemorrhage (ICH) is the most serious type of stroke, which results in a high disability or mortality rate. Therefore, accurate and rapid ICH region segmentation is of great significance for clinical diagnosis and treatment of ICH. In this paper, we focus on deep neural networks to automatically segment ICH regions. Firstly, we propose an encoder-decoder convolutional neural network (ED-Net) architecture to comprehensively utilizing both the low-level and high-level semantic information. Specifically, the encoder is used to extract multi-scale semantic feature information, while the decoder integrates them to form a unified ICH feature representation. Secondly, we introduce a synthetic loss function by paying more attention to the small ICH regions to overcome the data imbalanced problem. Thirdly, to improve the clinical adaptability of the proposed model, we collect 480 patient cases with ICH from four hospitals to construct a multi-center dataset, in which each case contains the first and review CT scans. In particular, CT scans of different patients are diverse, which greatly increases the difficulty of segmentation. Finally, we evaluate ED-Net on the multi-center ICH clinical dataset from different model parameters and different loss functions. We also compare the results of ED-Net with nine state-of-the-art methods in the literature. Both quantitative and visual results have shown that ED-Net outperforms other methods by providing more accurate and stable performance.

Image Deblurring Using Multi-Stream Bottom-Top-Bottom Attention Network and Global Information-Based Fusion and Reconstruction Network.

Image deblurring has been a challenging ill-posed problem in computer vision. Gaussian blur is a common model for image and signal degradation. The deep learning-based deblurring methods have attracted much attention due to their advantages over the traditional methods relying on hand-designed features. However, the existing deep learning-based deblurring techniques still cannot perform well in restoring the fine details and reconstructing the sharp edges. To address this issue, we have designed an effective end-to-end deep learning-based non-blind image deblurring algorithm. In the proposed method, a multi-stream bottom-top-bottom attention network (MBANet) with the encoder-to-decoder structure is designed to integrate low-level cues and high-level semantic information, which can facilitate extracting image features more effectively and improve the computational efficiency of the network. Moreover, the MBANet adopts a coarse-to-fine multi-scale strategy to process the input images to improve image deblurring performance. Furthermore, the global information-based fusion and reconstruction network is proposed to fuse multi-scale output maps to improve the global spatial information and recurrently refine the output deblurred image. The experiments were done on the public GoPro dataset and the realistic and dynamic scenes (REDS) dataset to evaluate the effectiveness and robustness of the proposed method. The experimental results show that the proposed method generally outperforms some traditional deburring methods and deep learning-based state-of-the-art deblurring methods such as scale-recurrent network (SRN) and denoising prior driven deep neural network (DPDNN) in terms of such quantitative indexes as peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) and human vision.

An Improved Faster R-CNN for UAV-Based Catenary Support Device Inspection

The catenary support device inspection is of crucial importance for ensuring safety and reliability of railway systems. At present, visual detection tasks of catenary support devices defect are performed by trained personnel based on the images taken periodically by industrial cameras installed on inspection vehicle in a limited period of time at midnight. However, the inspection mean is inappropriate for low efficiency and high cost. This paper presents a novel network based on unmanned aerial vehicle (UAV) images for catenary support device inspection and focuses on small object detection and the imbalanced dataset. With regards to the first aspect, based on a pyramid network structure, the improved Faster R-CNN consists of a top-down-top feature pyramid fusion structure, which heavily fuses high-level semantic information and low-level detail information. The feature map fusions of three different pooling scales are employed for improving detection accuracy of predicted bounding boxes. With regards to the second, we copy and paste the small proportion objects of dataset for avoiding category imbalance. Finally, quantitative and qualitative evaluations illustrate that the improved Faster-RCNN achieves better performance over the classic methods, yet remains convenient and efficient.

Blur detection via deep pyramid network with recurrent distinction enhanced modules

Blur detection aims to detect the regions where the image is blur and identifies the blurred regions accurately. Since the traditional hand-crafted feature based methods usually are not robust enough to handle various complex scenarios, the study of blur detection problem remains to be a challenging task in terms of the accuracy and effectiveness of blur separation. In this paper, instead of seeking a new hand-crafted feature often seen in common approaches, we present a new deep feature-learning method by using feature pyramid network via recurrent Distinction Enhanced Block modules to solve the blur detection problem. We show that the newly introduced Distinction Enhanced Block is able to merge the high-level semantic information with the low-level details effectively, while at the same time, it is capable to keep a desirable identification between the blurry and clear regions. We also develop a new boundary penalty term to refine the boundaries of our results, which leads to more accurate detection and segmentation, compared with current available results in literature. Furthermore, due to the lacking of public datasets for blur detection problems, we have established a new blur detection dataset which may enrich the experimental benchmarks for testing. In terms of efficiency associated with the competitive performance, in particular, our model is shown to have best-in-class evaluation speed, about 0.14 s to evaluate an input image in regular personal laptop. The performance evaluations on both CUHK dataset and our SZU-BD dataset validate that the proposed model outperforms various existing state-of-the-arts methods through commonly used metric indexes.

Augmenting Dementia Cognitive Assessment With Instruction-Less Eye-Tracking Tests.

Eye-tracking technology is an innovative tool that holds promise for enhancing dementia screening. In this work, we introduce a novel way of extracting salient features directly from the raw eye-tracking data of a mixed sample of dementia patients during a novel instruction-less cognitive test. Our approach is based on self-supervised representation learning where, by training initially a deep neural network to solve a pretext task using well-defined available labels (e.g. recognising distinct cognitive activities in healthy individuals), the network encodes high-level semantic information which is useful for solving other problems of interest (e.g. dementia classification). Inspired by previous work in explainable AI, we use the Layer-wise Relevance Propagation (LRP) technique to describe our network's decisions in differentiating between the distinct cognitive activities. The extent to which eye-tracking features of dementia patients deviate from healthy behaviour is then explored, followed by a comparison between self-supervised and handcrafted representations on discriminating between participants with and without dementia. Our findings not only reveal novel self-supervised learning features that are more sensitive than handcrafted features in detecting performance differences between participants with and without dementia across a variety of tasks, but also validate that instruction-less eye-tracking tests can detect oculomotor biomarkers of dementia-related cognitive dysfunction. This work highlights the contribution of self-supervised representation learning techniques in biomedical applications where the small number of patients, the non-homogenous presentations of the disease and the complexity of the setting can be a challenge using state-of-the-art feature extraction methods.