High-level Semantic Image Research Articles

Semantic Level Image Retrieval in Domain Adaptive Semantic Segmentation

The traditional domain adaptation task is not sufficient in mining inter domain image context. We propose a solution from the perspective of image semantic level retrieval. Image semantic level retrieval is based on the high-level semantics of images, retrieving the similarity or correspondence between images in terms of content and scene between domains. Our work focuses on the meaning, structure, and contextual information expressed by the target image, and retrieves the most relevant image from the source domain to improve the accuracy of domain adaptation. Furthermore, in response to the negative transfer brought about by image retrieval, we will introduce a category level weight adjustment module to further improve the accuracy of semantic level retrieval and weaken the negative impact caused by low precision categories. Finally, we demonstrated the effectiveness of semantic level image retrieval through experiments.

Image captioning using deep learning and python

In recent years, the confluence of computer vision and natural language processing, propelled by advancements in deep learning, has garnered significant interest. Among its notable applications, image captioning stands out, enabling computers to comprehend visual content through one or more sentences. This process entails not only identifying objects and scenes but also analyzing their attributes, states, and interrelations, culminating in the generation of meaningful descriptions encapsulating high-level image semantics. While inherently complex, image captioning has seen remarkable progress thanks to the efforts of numerous researchers. This paper offers a comprehensive review of three prominent image captioning methodologies leveraging deep neural networks: CNN-RNN, CNN-CNN, and Reinforcement-based frameworks. Each approach is accompanied by a detailed analysis of representative works, elucidating their respective contributions. Furthermore, evaluation metrics pertinent to these methods are discussed, followed by a synthesis of their advantages and primary challenges. Through this thorough examination, insights into the evolving landscape of image captioning are aimed to be provided, highlighting avenues for further exploration and innovation.

Hybrid CNN-transformer network for interactive learning of challenging musculoskeletal images

Background and objectivesSegmentation of regions of interest (ROIs) such as tumors and bones plays an essential role in the analysis of musculoskeletal (MSK) images. Segmentation results can help with orthopaedic surgeons in surgical outcomes assessment and patient's gait cycle simulation. Deep learning-based automatic segmentation methods, particularly those using fully convolutional networks (FCNs), are considered as the state-of-the-art. However, in scenarios where the training data is insufficient to account for all the variations in ROIs, these methods struggle to segment the challenging ROIs that with less common image characteristics. Such characteristics might include low contrast to the background, inhomogeneous textures, and fuzzy boundaries. Methodswe propose a hybrid convolutional neural network – transformer network (HCTN) for semi-automatic segmentation to overcome the limitations of segmenting challenging MSK images. Specifically, we propose to fuse user-inputs (manual, e.g., mouse clicks) with high-level semantic image features derived from the neural network (automatic) where the user-inputs are used in an interactive training for uncommon image characteristics. In addition, we propose to leverage the transformer network (TN) – a deep learning model designed for handling sequence data, in together with features derived from FCNs for segmentation; this addresses the limitation of FCNs that can only operate on small kernels, which tends to dismiss global context and only focus on local patterns. ResultsWe purposely selected three MSK imaging datasets covering a variety of structures to evaluate the generalizability of the proposed method. Our semi-automatic HCTN method achieved a dice coefficient score (DSC) of 88.46 ± 9.41 for segmenting the soft-tissue sarcoma tumors from magnetic resonance (MR) images, 73.32 ± 11.97 for segmenting the osteosarcoma tumors from MR images and 93.93 ± 1.84 for segmenting the clavicle bones from chest radiographs. When compared to the current state-of-the-art automatic segmentation method, our HCTN method is 11.7%, 19.11% and 7.36% higher in DSC on the three datasets, respectively. ConclusionOur experimental results demonstrate that HCTN achieved more generalizable results than the current methods, especially with challenging MSK studies.

Hierarchical ConViT with Attention-Based Relational Reasoner for Visual Analogical Reasoning

Raven’s Progressive Matrices (RPMs) have been widely used to evaluate the visual reasoning ability of humans. To tackle the challenges of visual perception and logic reasoning on RPMs, we propose a Hierarchical ConViT with Attention-based Relational Reasoner (HCV-ARR). Traditional solution methods often apply relatively shallow convolution networks to visually perceive shape patterns in RPM images, which may not fully model the long-range dependencies of complex pattern combinations in RPMs. The proposed ConViT consists of a convolutional block to capture the low-level attributes of visual patterns, and a transformer block to capture the high-level image semantics such as pattern formations. Furthermore, the proposed hierarchical ConViT captures visual features from multiple receptive fields, where the shallow layers focus on the image fine details while the deeper layers focus on the image semantics. To better model the underlying reasoning rules embedded in RPM images, an Attention-based Relational Reasoner (ARR) is proposed to establish the underlying relations among images. The proposed ARR well exploits the hidden relations among question images through the developed element-wise attentive reasoner. Experimental results on three RPM datasets demonstrate that the proposed HCV-ARR achieves a significant performance gain compared with the state-of-the-art models. The source code is available at: https://github.com/wentaoheunnc/HCV-ARR.

AttentionGAN: Unpaired Image-to-Image Translation Using Attention-Guided Generative Adversarial Networks.

State-of-the-art methods in the image-to-image translation are capable of learning a mapping from a source domain to a target domain with unpaired image data. Though the existing methods have achieved promising results, they still produce visual artifacts, being able to translate low-level information but not high-level semantics of input images. One possible reason is that generators do not have the ability to perceive the most discriminative parts between the source and target domains, thus making the generated images low quality. In this article, we propose a new Attention-Guided Generative Adversarial Networks (AttentionGAN) for the unpaired image-to-image translation task. AttentionGAN can identify the most discriminative foreground objects and minimize the change of the background. The attention-guided generators in AttentionGAN are able to produce attention masks, and then fuse the generation output with the attention masks to obtain high-quality target images. Accordingly, we also design a novel attention-guided discriminator which only considers attended regions. Extensive experiments are conducted on several generative tasks with eight public datasets, demonstrating that the proposed method is effective to generate sharper and more realistic images compared with existing competitive models. The code is available at https://github.com/Ha0Tang/AttentionGAN.

Learning Attribute-guided Fashion Similarity with Spatial and Channel Attention

ABSTRACT Fashion image retrieval is one of the important services of e-commerce platforms, and it is also the basis of various fashion-related AI applications. Studies have shown that in a multi-modal environment (images + attribute labels), embedding items into specific attribute spaces can support more fine-grained similarity measures, which is especially suitable for fashion retrieval tasks. In this paper, we propose an attention-based attribute-guided similarity learning network (AttnFashion) for fashion image retrieval. The core of this network is an attribute-guided spatial attention module and an attribute-guided channel attention module, which correspond to the mapping between attributes and image regions, and the mapping between attributes and high-level image semantics, respectively. To make these two modules interact deeply, we design a parallel structure that allows them to share attribute embeddings and guide each other to extract specific features, which also helps to reduce the network parameters of the attention modules. An adaptive feature fusion strategy is proposed to synthesise the features extracted by the two modules. Extensive experiments show that the proposed AttnFashion performs better than current competitive networks in the field of fine-grained attribute-based fashion retrieval.

Hyper-fusion network for semi-automatic segmentation of skin lesions.

Segmentation of skin lesions is an important step for imaging-based clinical decision support systems. Automatic skin lesion segmentation methods based on fully convolutional networks (FCNs) are regarded as the state-of-the-art for accuracy. When there are, however, insufficient training data to cover all the variations in skin lesions, where lesions from different patients may have major differences in size/shape/texture, these methods failed to segment the lesions that have image characteristics, which are less common in the training datasets. FCN-based semi-automatic segmentation methods, which fuse user-inputs with high-level semantic image features derived from FCNs offer an ideal complement to overcome limitations of automatic segmentation methods. These semi-automatic methods rely on the automated state-of-the-art FCNs coupled with user-inputs for refinements, and therefore being able to tackle challenging skin lesions. However, there are a limited number of FCN-based semi-automatic segmentation methods and all these methods focused on 'early-fusion', where the first few convolutional layers are used to fuse image features and user-inputs and then derive fused image features for segmentation. For early-fusion based methods, because the user-input information can be lost after the first few convolutional layers, consequently, the user-input information will have limited guidance and constraint in segmenting the challenging skin lesions with inhomogeneous textures and fuzzy boundaries. Hence, in this work, we introduce a hyper-fusion network (HFN) to fuse the extracted user-inputs and image features over multiple stages. We separately extract complementary features which then allows for an iterative use of user-inputs along all the fusion stages to refine the segmentation. We evaluated our HFN on three well-established public benchmark datasets - ISBI Skin Lesion Challenge 2017, 2016 and PH2 - and our results show that the HFN is more accurate and generalizable than the state-of-the-art methods, in particular with challenging skin lesions.

Self-Supervised Learning With Prediction of Image Scale and Spectral Order for Hyperspectral Image Classification

In recent years, Convolutional Neural Networks (CNNs) have achieved great success in hyperspectral image classification attributed to their unparalleled capacity to extract the local information. However, to successfully learn the high-level semantic image features, they always require massive amounts of manually labeled data during the training process, which is expensive, scarce, and impractical, and severely hinders the improvement of supervised deep learning methods. To alleviate these burdens, we present Self-Supervised Learning methods for hyperspectral image classification by a pre-training model using extensive unlabeled data and fine-tuning the hyperspectral image target classification. In this paper, we propose a new method for learning image characteristics by training a CNN to recognize the image scale that is applied to the hyperspectral images (HSIs). In addition, we propose a multi-pretext task method to learn stable and good feature representations combing two different pretext task methods and contrastive loss function. We evaluate the proposed methods in Self-Supervised Learning benchmarks on four benchmark HSIs datasets. The experiment results demonstrate that the proposed methods outperform the traditional supervised deep learning methods when large amounts of unlabeled HSIs data are used. Moreover, it demonstrates that the Self-Supervised Learning method is promising to alleviate dependence on manually labeled data of hyperspectral image classification. Finally, our research contributes to the creation and refinement of Self-Supervised Learning methods for pretextual tasks within the HSIs community.

Sensitivity Detection of Retinal Nerve Fiber Layer in Glaucoma Based on High Level Semantic Image Fusion Algorithm

Glaucoma is currently recognized as a multifactorial, persistent and degenerative retinal disease. It mainly causes the loss of function and death of retinal ganglion cells in the optic nerve head area, and eventually leads to visual loss and blindness. Aiming at the segmentation and sensitivity detection of retinal nerve fiber layer (RNFL) in glaucoma, this paper mainly studies it based on high-level semantic image fusion algorithm. Firstly, the feature extracted by high-level semantic image fusion technology is used to train random forest classifier to segment retina to get rough position of the boundary of nerve fiber layer, then the first step of rough boundary is refined by boundary tracking algorithm to get the final result of retinal layer segmentation. In this algorithm, random forest classifier is used to find the boundary of single pixel width between layers of retina, and 12 features are used to train random forest classifier. Among them, relative gray feature and neighborhood feature can solve the problem of large segmentation error of uneven gray. By using high-level semantic image technology, the mean value and gradient features are extracted under multi-scale, and the relative gray difference features and neighborhood features are introduced, then the features are trained by random forest classifier. The trained classifier gives different labels to the unclassified features, and finally successfully segments the lower boundary of the retinal nerve fiber layer, which solves the sensitive segmentation and detection problem of the retinal nerve fiber layer with uneven pixel gray.

Feature Extraction and Retrieval of Ecommerce Product Images Based on Image Processing

The new retail is an industry featured by online ecommerce. One of the key techniques of the industry is the product identification based on image processing. This technique has an important business application value, because it is capable of improving the retrieval efficiency of products and the level of information supervision. To acquire high-level semantics of images and enhance the retrieval effect of products, this paper explores the feature extraction and retrieval of ecommerce product images based on image processing. The improved Fourier descriptor was innovatively into a metric learning-based product image feature extraction network, and the attention mechanism was introduced to realize accurate retrieval of product images. Firstly, the authors detailed how to acquire the product contour and the axis with minimum moment of inertia, and then extracted the shape feature of products. Next, a feature extraction network was established based on the metric learning supervision, which is capable of obtaining distinctive feature, and thus realized the extraction of distinctive and classification features of products. Finally, the authors expounded on the product image retrieval method based on cluster attention neural network. The effectiveness of our method was confirmed through experiments. The research results provide a reference for feature extraction and retrieval in other fields of image processing.

Image captioning Bot

Image captioning means automatically generating a caption for an image with the development of deep learning, the combination of computer vision and natural language process has caught great attention in the last few years. Image captioning is a representative of this filed, which makes the computer learn to use one or more sentences to understand the visual content of an image. The meaningful description generation process of highlevel image semantics requires not only the recognition of the object and the scene, but the ability of analyzing the state, the attributes and the relationship among these objects. Neural network based methods are further divided into subcategories based on the specific framework they use. Each subcategory of neural network based methods are discussed in detail. After that, state of the art methods are compared on benchmark datasets. Following that, discussions on future research directions are presented.

Multiview Active Learning for Scene Classification with High-Level Semantic-Based Hypothesis Generation

Multiview active learning (MVAL) is a technique which can result in a large decrease in the size of the version space than traditional active learning and has great potential applications in large-scale data analysis. This paper made research on MVAL-based scene classification for helping the computer accurately understand diverse and complex environments macroscopically, which has been widely used in many fields such as image retrieval and autonomous driving. The main contribution of this paper is that different high-level image semantics are used for replacing the traditional low-level features to generate more independent and diverse hypotheses in MVAL. First, our algorithm uses different object detectors to achieve local object responses in the scenes. Furthermore, we design a cascaded online LDA model for mining the theme semantic of an image. The experimental results demonstrate that our proposed theme modeling strategy fits the large-scale data learning, and our MVAL algorithm with both high-level semantic views can achieve significant improvement in the scene classification than traditional active learning-based algorithms.

Quality Evaluation for Image Retargeting With Instance Semantics

To meet the ever-increasing demand for devices with diversified displays, image retargeting has become a prevalent technique for adaptive image resizing. In practice, the retargeting operation inevitably causes impairments in the images; thus, image retargeting quality assessment (IRQA) is urgently needed and, can be used to guide algorithm optimization, selection and design. Unlike traditional image quality assessment, image retargeting introduces geometric distortions, which typically affect high-level image semantics. With this motivation, this paper presents a quality evaluation model for image retargeting based on INstance SEMantics (INSEM). Considering that the human visual system (HVS) perceives images highly dependent on apprehensible areas and that impairments in image retargeting mainly degrade the salient instances, an image instance is utilized as the basic semantic unit, and a top-down method is devised to extract instance-level semantic features for IRQA. In addition, taking into account the influence of semantic categories on the perception of retargeting quality, we further propose Semantic-based self-adaptive pooling (SSAP) to integrate instance-based semantic features. Finally, global features are incorporated to generate quality scores that are more consistent with people's perceptions. Extensive experiments and comparisons of three public databases, in terms of both intradatabase and cross-database settings, demonstrate the superiority of the proposed metric over state-of-the-art methods.

Saliency from High-Level Semantic Image Features

Top-down semantic information is known to play an important role in assigning saliency. Recently, large strides have been made in improving state-of-the-art semantic image understanding in the fields of object detection and semantic segmentation. Therefore, since these methods have now reached a high-level of maturity, evaluation of the impact of high-level image understanding on saliency estimation is now feasible. We propose several saliency features which are computed from object detection and semantic segmentation results. We combine these features with a standard baseline method for saliency detection to evaluate their importance. Experiments demonstrate that the proposed features derived from object detection and semantic segmentation improve saliency estimation significantly. Moreover, they show that our method obtains state-of-the-art results on (FT, ImgSal, and SOD datasets) and obtains competitive results on four other datasets (ECSSD, PASCAL-S, MSRA-B, and HKU-IS).

A review on visual content-based and users’ tags-based image annotation: methods and techniques

In the current era of digital communication, the use of images is growing exponentially since they are one of the best ways of expressing, sharing and memorizing knowledge. In fact, images can be used in various real-world applications, like biology, medical diagnosis, space research, remote sensing, etc. However, finding the most relevant images that meet the users??? needs is a challenging task, especially when the search is performed over gigantic amounts of images. This has led to the emergence of several image retrieval studies during the past two decades. Typically, research studies in this area were focused on the Content-based Image Retrieval (CBIR). However, extensive research have proved that there is a ???semantic gap??? between the visual information captured by the imaging devices and the image semantics understandable by humans. As an alternative, researchers??? efforts have been oriented towards the Text-based Image Retrieval (TBIR). Indeed, TBIR is a typical method that helps bridge the issue of ???semantic gap??? between the low-level image features and the high-level image semantics. Its policy consists in associating textual descriptions with the images, which constitute the focus of the research queries later on. In this paper, we analyze various image annotation methods, namely: Visual Content-based and Users??? Tags-based Image Annotation Methods. In particular, we focus on the visual content-based image annotation techniques since they are one of the dynamic research fields nowadays.

Face image recognition based on convolutional neural network

With the continuous progress of The Times and the development of technology, the rise of network social media has also brought the explosive growth of image data. As one of the main ways of People's Daily communication, image is widely used as a carrier of communication because of its rich content, intuitive and other advantages. Image recognition based on convolution neural network is the first application in the field of image recognition. A series of algorithm operations such as image eigenvalue extraction, recognition and convolution are used to identify and analyze different images. The rapid development of artificial intelligence makes machine learning more and more important in its research field. Use algorithms to learn each piece of data and predict the outcome. This has become an important key to open the door of artificial intelligence. In machine vision, image recognition is the foundation, but how to associate the low-level information in the image with the high-level image semantics becomes the key problem of image recognition. Predecessors have provided many model algorithms, which have laid a solid foundation for the development of artificial intelligence and image recognition. The multi-level information fusion model based on the VGG16 model is an improvement on the fully connected neural network. Different from full connection network, convolutional neural network does not use full connection method in each layer of neurons of neural network, but USES some nodes for connection. Although this method reduces the computation time, due to the fact that the convolutional neural network model will lose some useful feature information in the process of propagation and calculation, this paper improves the model to be a multi-level information fusion of the convolution calculation method, and further recovers the discarded feature information, so as to improve the recognition rate of the image. VGG divides the network into five groups (mimicking the five layers of AlexNet), yet it USES 3∗3 filters and combines them as a convolution sequence. Network deeper DCNN, channel number is bigger. The recognition rate of the model was verified by 0RL Face Database, BioID Face Database and CASIA Face Image Database.

Multi-source Multi-level Attention Networks for Visual Question Answering

In recent years, Visual Question Answering (VQA) has attracted increasing attention due to its requirement on cross-modal understanding and reasoning of vision and language. VQA is proposed to automatically answer natural language questions with reference to a given image. VQA is challenging, because the reasoning process on a visual domain needs a full understanding of the spatial relationship, semantic concepts, as well as the common sense for a real image. However, most existing approaches jointly embed the abstract low-level visual features and high-level question features to infer answers. These works have limited reasoning ability due to the lack of modeling of the rich spatial context of regions, high-level semantics of images, and knowledge across multiple sources. To solve the challenges, we propose multi-source multi-level attention networks for visual question answering that can benefit both spatial inferences by visual attention on context-aware region representation and reasoning by semantic attention on concepts as well as external knowledge. Indeed, we learn to reason on image representation by question-guided attention at different levels across multiple sources, including region and concept level representation from image source as well as sentence level representation from the external knowledge base. First, we encode region-based middle-level outputs from Convolutional Neural Networks (CNNs) into spatially embedded representation by a multi-directional two-dimensional recurrent neural network and, further, locate the answer-related regions by Multiple Layer Perceptron as visual attention. Second, we generate semantic concepts from high-level semantics in CNNs and select those question-related concepts as concept attention. Third, we query semantic knowledge from the general knowledge base by concepts and selected question-related knowledge as knowledge attention. Finally, we jointly optimize visual attention, concept attention, knowledge attention, and question embedding by a softmax classifier to infer the final answer. Extensive experiments show the proposed approach achieved significant improvement on two very challenging VQA datasets.

Image Captioning Based on Deep Neural Networks

With the development of deep learning, the combination of computer vision and natural language process has aroused great attention in the past few years. Image captioning is a representative of this filed, which makes the computer learn to use one or more sentences to understand the visual content of an image. The meaningful description generation process of high level image semantics requires not only the recognition of the object and the scene, but the ability of analyzing the state, the attributes and the relationship among these objects. Though image captioning is a complicated and difficult task, a lot of researchers have achieved significant improvements. In this paper, we mainly describe three image captioning methods using the deep neural networks: CNN-RNN based, CNN-CNN based and Reinforcement-based framework. Then we introduce the representative work of these three top methods respectively, describe the evaluation metrics and summarize the benefits and major challenges.

A learning automata framework based on relevance feedback for content-based image retrieval

The need for efficient image browsing and searching motivates the use of Content-Based Image Retrieval (CBIR) systems. However, they suffer from a big gap between high-level image semantics and low-level features. So, a learning process to reduce the gap seems quite useful. This paper presents a novel Learning Automata (LA)-based approach to improve the CBIR systems. Distributed Learning Automata (DLA) is used in this work to learn the relevant images from textual query feedbacks of the users. Subsequently, the retrieved images are ranked according to the learning outcome and similarity measure. In this study, the similarity between images is evaluated based on two color descriptors: the global color histogram and local color auto-correlogram. A thorough observation and comparison of these color descriptors performances are performed with different color spaces and also with various similarity measures. Experimental results on two publicly available databases demonstrate that the performance of the proposed CBIR system after each round is improved and the system could retrieve images compatible with the users’ perception.

Semantic Classification of High-Resolution Remote-Sensing Images Based on Mid-level Features

With the resolution improvement of the remote-sensing images, more details are shown clearly. The challenge that comes along is how to boost the relatively low classification accuracy caused by using pixel-based image classification approaches and low-level visual structure. The low-level features (LLF) may not well describe the image due to the semantic gap between low-level visual features and high-level semantics of images. The bag-of-visual-words (BOV) model which generates mid-level features was proposed to bridge the two levels. However, it generally neglects the context information between local patches. In this paper, an object-oriented semantic classification algorithm that combines BOV with the optimal segmentation scale is presented. In this algorithm, BOV addresses the problem of the representation of mid-level for scenes, while the optimal segmentation scale intends to overcome the defect of conventional BOV in lacking of relationship between image patches and to give more thorough description. The object-based BOV is presented to construct mid-level representations for object description instead of LLF, and histogram intersection kernel (HIK) is introduced in support vector machine (SVM) for classification. The experiments conducted on three datasets testify the superiority of the proposed algorithm.