Single Image Feature Research Articles

Integrating temporal-aggregated satellite image with multi-sensor image fusion for seasonal land-cover mapping of Shilansha watershed, rift valley basin of Ethiopia

Accurate land-cover mapping in regions with frequent cloud-cover and rapidly changing agricultural land cover by crop growth cycles cannot be guaranteed by use of single sensor images, or an image from a single-acquisition-date. This study addressed these challenges by applying temporal-aggregation of single sensor image features that is integrated with multi-sensor image fusion. Results of land-cover classification target fallow, growing, and harvest/post-harvest agricultural seasons. Satellite based features used were frequency bands of Sentinel-1 (S1) and Sentinel-2 (S2), including vegetation indices (VIs) and biophysical variables (BPVs). Temporal aggregation improved classification accuracy. The single-acquisition-date S2 image, overall accuracy (OA) ranged from 0.81 to 0.85, increased to 0.86 to 0.87 after temporal-aggregation. Meanwhile, for single-acquisitions of S1, OA ranged from 0.44 to 0.79 increased to 0.6 to 0.86 across respective seasons. Fusing temporally aggregated S1 and S2 image features including VIs and BPVs increased OA up to 0.90. Selecting 11, 8, and 10 out of 18 optimum numbers of features for fallow, growing, and harvest/post-harvest seasons respectively improved OA by 3%, 2%, and 1.86%. PCA fusion of the temporally aggregated best performing feature set enhanced harvest/post-harvest season, fallow, and growing seasons with OA of 0.98, 0.96 and 0.94 respectively. Accuracy was enhanced when selecting different best performing feature set for the three seasons. The study enhanced knowledge of advanced remote sensing for agricultural land cover mapping, with practical implications of land monitoring and management.

Urban Perception Assessment from Street View Images Based on a Multifeature Integration Encompassing Human Visual Attention

Urban perception is a multifaceted process, spanning from vision to cognitive interpretation. Earlier studies on urban perception assessment have primarily concentrated on the analysis of graphic features of street view images such as color or semantic elements, neglecting the vital human sensory fixation features in cognitive processes. This study leveraged eye-tracking technology to accumulate human visual attention data during emotion scoring of street view images. Additionally, a deep learning network model was proposed to forecast human visual attention areas by assimilating environmental features such as image color, depth, and semantic elements, and subsequently amalgamating the aforementioned environmental features and visual attention to establish a perception prediction model, capable of predicting urban perception on a larger scale. Finally, the reliability and superiority of the urban perception prediction method were confirmed through ablation experiments and bias analysis. On the one hand, the method enhances the prediction accuracy of urban perception; on the other, it investigates the comprehensive influence of multiple features on human subjective emotions from the external environment perspective and individual perspective. This approach transcends the former limited reliance on single-image features, providing a novel and scientific approach for assessing urban perception.

Multi-level feature extraction and reconstruction for 3D MRI image super-resolution

Magnetic resonance imaging (MRI) is an essential radiology technique in clinical diagnosis, but its spatial resolution may not suffice to meet the growing need for precise diagnosis due to hardware limitations and thicker slice thickness. Therefore, it is crucial to explore suitable methods to increase the resolution of MRI images. Recently, deep learning has yielded many impressive results in MRI image super-resolution (SR) reconstruction. However, current SR networks mainly use convolutions to extract relatively single image features, which may not be optimal for further enhancing the quality of image reconstruction. In this work, we propose a multi-level feature extraction and reconstruction (MFER) method to restore the degraded high-resolution details of MRI images. Specifically, to comprehensively extract different types of features, we design the triple-mixed convolution by leveraging the strengths and uniqueness of different filter operations. For the features of each level, we then apply deconvolutions to upsample them separately at the tail of the network, followed by the feature calibration of spatial and channel attention. Besides, we also use a soft cross-scale residual operation to improve the effectiveness of parameter optimization. Experiments on lesion-free and glioma datasets indicate that our method obtains superior quantitative performance and visual effects when compared with state-of-the-art MRI image SR methods.

Super-resolution reconstruction of binocular image based on multi-level fusion attention network

Super-resolution reconstruction of binocular image based on multi-level fusion attention network

3D-FVS: construction and application of three-dimensional fundus vascular structure model based on single image features

BackgroundFundus microvasculature may be visually observed by ophthalmoscope and has been widely used in clinical practice. Due to the limitations of available equipment and technology, most studies only utilized the two-dimensional planar features of the fundus microvasculature.MethodsThis study proposed a novel method for establishing the three-dimensional fundus vascular structure model and generating hemodynamic characteristics based on a single image. Firstly, the fundus vascular are segmented through our proposed network framework. Then, the length and width of vascular segments and the relationship among the adjacent segments are collected to construct the three-dimensional vascular structure model. Finally, the hemodynamic model is generated based on the vascular structure model, and highly correlated hemodynamic features are selected to diagnose the ophthalmic diseases.ResultsIn fundus vascular segmentation, the proposed network framework obtained 98.63% and 97.52% on Area Under Curve (AUC) and accuracy respectively. In diagnosis, the high correlation features extracted based on the proposed method achieved 95% on accuracy.ConclusionsThis study demonstrated that hemodynamic features filtered by relevance were essential for diagnosing retinal diseases. Additionally, the method proposed also outperformed the existing models on the levels of retina vessel segmentation. In conclusion, the proposed method may represent a novel way to diagnose retinal related diseases, which can analysis two-dimensional fundus pictures by extracting heterogeneous three-dimensional features.

Imaging flow cytometry: a primer.

Imaging flow cytometry combines the high throughput nature of flow cytometry with the advantages of single cell image acquisition associated with microscopy. The measurement of large numbers of features from the resulting images provides rich datasets which have resulted in a wide range of novel biomedical applications. In this primer we discuss the typical imaging flow instrumentation, the form of data acquired and the typical analysis tools that can be applied to this data. Using examples from the literature we discuss the progression of the analysis methods that have been applied to imaging flow cytometry data. These methods start from the use of simple single image features and multiple channel gating strategies, followed by the design and use of custom features for phenotype classification, through to powerful machine and deep learning methods. For each of these methods, we outline the processes involved in analyzing typical datasets and provide details of example applications. Finally we discuss the current limitations of imaging flow cytometry and the innovations which are addressing these challenges.

A hybrid spatial–temporal deep learning architecture for lane detection

AbstractAccurate and reliable lane detection is vital for the safe performance of lane‐keeping assistance and lane departure warning systems. However, under certain challenging circumstances, it is difficult to get satisfactory performance in accurately detecting the lanes from one single image as mostly done in current literature. Since lane markings are continuous lines, the lanes that are difficult to be accurately detected in the current single image can potentially be better deduced if information from previous frames is incorporated. This study proposes a novel hybrid spatial–temporal (ST) sequence‐to‐one deep learning architecture. This architecture makes full use of the ST information in multiple continuous image frames to detect the lane markings in the very last frame. Specifically, the hybrid model integrates the following aspects: (a) the single image feature extraction module equipped with the spatial convolutional neural network; (b) the ST feature integration module constructed by ST recurrent neural network; (c) the encoder–decoder structure, which makes this image segmentation problem work in an end‐to‐end supervised learning format. Extensive experiments reveal that the proposed model architecture can effectively handle challenging driving scenes and outperforms available state‐of‐the‐art methods.

An Efficient Deep Learning-based Content-based Image Retrieval Framework

The use of massive image databases has increased drastically over the few years due to evolution of multimedia technology. Image retrieval has become one of the vital tools in image processing applications. Content-Based Image Retrieval (CBIR) has been widely used in varied applications. But, the results produced by the usage of a single image feature are not satisfactory. So, multiple image features are used very often for attaining better results. But, fast and effective searching for relevant images from a database becomes a challenging task. In the previous existing system, the CBIR has used the combined feature extraction technique using color auto-correlogram, Rotation-Invariant Uniform Local Binary Patterns (RULBP) and local energy. However, the existing system does not provide significant results in terms of recall and precision. Also, the computational complexity is higher for the existing CBIR systems. In order to handle the above mentioned issues, the Gray Level Co-occurrence Matrix (GLCM) with Deep Learning based Enhanced Convolution Neural Network (DLECNN) is proposed in this work. The proposed system framework includes noise reduction using histogram equalization, feature extraction using GLCM, similarity matching computation using Hierarchal and Fuzzy c- Means (HFCM) algorithm and the image retrieval using DLECNN algorithm. The histogram equalization has been used for computing the image enhancement. This enhanced image has a uniform histogram. Then, the GLCM method has been used to extract the features such as shape, texture, colour, annotations and keywords. The HFCM similarity measure is used for computing the query image vector's similarity index with every database images. For enhancing the performance of this image retrieval approach, the DLECNN algorithm is proposed to retrieve more accurate features of the image. The proposed GLCM+DLECNN algorithm provides better results associated with high accuracy, precision, recall, f-measure and lesser complexity. From the experimental results, it is clearly observed that the proposed system provides efficient image retrieval for the given query image.

Deep Stereoscopic Image Super-Resolution via Interaction Module

Deep learning-based methods have achieved remarkable performance in single image super-resolution. However, these methods cannot be effectively applied in stereoscopic image super-resolution without considering the characteristics of stereoscopic images. In this article, an interaction module-based stereoscopic image super-resolution network (IMSSRnet) is proposed to effectively utilize the correlation information in stereoscopic images. The key insight of the network lies with how to explore the complementary information of one view to help the reconstruction of another view. Thus, an interaction module is designed to acquire the enhanced features by utilizing complementary information between different views. Specifically, the interaction module is composed of a series of interaction units with a residual structure. In addition, the single image features of left and right views are obtained by a spatial feature extraction module, which can be realized by any existing single image super-resolution models. In order to obtain high-quality stereoscopic images, a gradient loss is introduced to preserve the texture details in a view, and a disparity loss is developed to constrain the disparity relationship between different views. Experimental results demonstrate that the proposed method achieves a promising performance and outperforms the state-of-the-art methods.

Malware homology determination using visualized images and feature fusion.

The family homology determination of malware has become a research hotspot as the number of malware variants are on the rise. However, existing studies on malware visualization only determines homology based on the global structure features of executable, which leads creators of some malware variants with the same structure intentionally set to misclassify them as the same family. We sought to develop a homology determination method using the fusion of global structure features and local fine-grained features based on malware visualization. Specifically, the global structural information of the malware executable file was converted into a bytecode image, and the opcode semantic information of the code segment was extracted by the n-gram feature model to generate an opcode image. We also propose a dual-branch convolutional neural network, which features the opcode image and bytecode image as the final family classification basis. Our results demonstrate that the accuracy and F-measure of family homology classification based on the proposed scheme are 99.05% and 98.52% accurate, respectively, which is better than the results from a single image feature or other major schemes.

Depth-aware salient object segmentation

Depth-aware salient object segmentation

Annular Spatial Pyramid Mapping and Feature Fusion-Based Image Coding Representation and Classification

Conventional image classification models commonly adopt a single feature vector to represent informative contents. However, a single image feature system can hardly extract the entirety of the information contained in images, and traditional encoding methods have a large loss of feature information. Aiming to solve this problem, this paper proposes a feature fusion-based image classification model. This model combines the principal component analysis (PCA) algorithm, processed scale invariant feature transform (P-SIFT) and color naming (CN) features to generate mutually independent image representation factors. At the encoding stage of the scale-invariant feature transform (SIFT) feature, the bag-of-visual-word model (BOVW) is used for feature reconstruction. Simultaneously, in order to introduce the spatial information to our extracted features, the rotation invariant spatial pyramid mapping method is introduced for the P-SIFT and CN feature division and representation. At the stage of feature fusion, we adopt a support vector machine with two kernels (SVM-2K) algorithm, which divides the training process into two stages and finally learns the knowledge from the corresponding kernel matrix for the classification performance improvement. The experiments show that the proposed method can effectively improve the accuracy of image description and the precision of image classification.

Integration of spectra and image features of Vis/NIR hyperspectral imaging for prediction of deoxynivalenol contamination in whole wheat flour

This study was conducted to investigate the feasibility of combining spectral with image information for prediction of deoxynivalenol (DON) contamination in whole wheat flour by Vis/NIR hyperspectral imaging. A total of 195 samples were collected from 13 cities in China with DON contents ranged from 0.0 mg kg−1 to 6.233 mg kg−1. Characteristic spectral variables and color parameters of samples were extracted from hyperspectral images and were integrated to chemometric analysis. Principal component analysis (PCA) indicated a certain clustering tendency between acceptable samples and infected samples when setting 1.0 mg kg−1 as the cutoff value. Subsequently, linear discriminant analysis (LDA) modeling based on spectral and color fusion features obtained correct classified rate of 96.92% for discrimination of acceptable and infected samples, which was much higher than single spectral or image feature. Microscopic analysis indicated that the higher the DON content, the looser the binding of wheat starch granules. However, quantification of DON was not satisfied by partial least squares regression (PLSR) modeling (Rp = 0.691, RMSEP = 0.707 mg kg−1). Overall, these findings verify the potential of hyperspectral imaging for rapid discrimination of DON contamination in whole wheat flour.

DE-Ada*: A novel model for breast mass classification using cross-modal pathological semantic mining and organic integration of multi-feature fusions

Computer-aided breast mass classification is an effective and widely used technology to assist pathologists in formulating clinical diagnoses and improving working efficiencies. Existing studies usually use a single image feature to perform breast mass classification. Herein, we propose a simple, yet effective, model called the DE-Ada*, which is an organic integration of multi-feature fusions, for breast mass classification. Firstly, we extract a set of complementary features, namely, scale-invariant feature transform (SIFT), GIST, histogram of oriented gradient (HOG), local binary pattern (LBP), residual network (ResNet), densely connected convolutional networks (DenseNet), and visual geometry group (VGG), to characterize mammograms from diverse perspectives. We attempt to mine the cross-modal pathological semantics among these features and complete their early fusion. The dynamic weight of any feature or cross-modal pathological semantics is computed and utilized to complete mid-level feature fusion. Finally, we design two voting-based ensemble learning strategies to implement late feature fusion. Our experiments demonstrate that the DE-Ada* model outperforms baselines on two well-known mammographic datasets. Our model encourages the use of cross-modal pathological semantics to deal with the overfitting problem.

Improving the generalization performance of deep networks by dual pattern learning with adversarial adaptation

In this paper, we present a dual pattern learning network architecture with adversarial adaptation (DPLAANet). Unlike conventional networks, the proposed network has two input branches and two loss functions. This architecture forces the network to learn robust features by analysing dual inputs. The dual input structure allows the network to have a considerably large number of image pairs, which can help address the overfitting issue due to limited training data. In addition, we propose to associate the two input branches with two random interest values during training. As a stochastic regularization technique, this method can improve the generalization performance. Moreover, we introduce to use the adversarial training approach to reduce the domain difference between fused image features and single image features. Extensive experiments on CIFAR-10, CIFAR-100, FI-8, the Google commands dataset, and MNIST demonstrate that our DPLAANets exhibit better performance than the baseline networks. The experimental results on subsets of CIFAR-10, CIFAR-100, and MNIST demonstrate that DPLAANets have a good generalization performance on small datasets. The proposed architecture can be easily extended to have more than two input branches. The experimental results on subsets of MNIST show that the architecture with three branches outperforms two branches when the training set is extremely small.

An End-to-End Mammogram Diagnosis: A New Multi-Instance and Multiscale Method Based on Single-Image Feature

Mammography is the most common modality used in breast cancer detection. Most diagnostic mammography studies, however, are based on single-image training with little attention to the fact that the size of breast lesions varies significantly and the overall condition of the breast from different views. Therefore, the methodology is not in accord with the clinical requirement. To overcome this problem, we propose a new end-to-end method for mammographic diagnosis. As part of this process, we construct a data set of patients from West China Hospital to validate the new method. Furthermore, a multiscale module is proposed for the acquisition of complex breast features in a single image, enabling the screening of unique features in variably sized lesions. Finally, a multi-instance module is proposed for realistic hospital requirements to calculate the contribution of each mammogram in reaching the final diagnosis. Guidance by the single-image features can ameliorate the problem of weak one-case labeling. The new method yielded both a public data set and a realistic hospital data set.

Learning Recurrent 3D Attention for Video-Based Person Re-Identification

In this paper, we propose to learn recurrent 3D attention (A3D) for video-based person re-identification. Attention model plays a key role in both spatial and temporal domains for video representation. Most existing methods apply spatial attention model to extract feature from a single image and aggregate image features with attentive temporal pooling or RNN. However, the inherent consistencies and correlations between spatial and temporal clues are not leveraged. Our A3D method aims to utilize the joint constraints of temporal and spatial attentions to enhance the robustness of attention model. Towards this goal, we treat the pedestrian video as a unified 3D bin where the temporal domain is denoted as an additional dimension. Then we develop an attention agent to iteratively select the locations of the salient spatial-temporal parts in the 3D bin. In addition, we formulate our sequential 3D attention learning as a Markov Decision Process and train the representation network and attention detector with the policy gradient method in an end-to-end manner. We evaluate the proposed method on three challenging datasets including iLIDS-VID, PRID-2011 and the large-scale MARS dataset, and consistently improve the performance in comparison with the state-of-the-art methods.

Query-Adaptive Remote Sensing Image Retrieval Based on Image Rank Similarity and Image-to-Query Class Similarity

Many image features have been proposed for image retrieval; hence, effectively fusing these features to alleviate the large variation in performance among image queries when using single image features has become a major challenge in remote sensing (RS) image retrieval. Because high-resolution remote sensing images have abundant and complex visual contents, accurately measuring the similarity between two images is another important problem. To address these challenges, we propose a novel RS image retrieval method that uses query-adaptive feature weights to fuse features and utilizes two image similarities to improve retrieval performance. First, we use the image rank similarity, which measures the similarity between two images according to their corresponding top-m image lists from a reference image collection, to calculate the similarity of each feature between a query image and each retrieved image. Then, we assign a weight to each feature to fuse these features via our query-adaptive weighting method. Finally, we take the query image and its neighborhood set selected from the retrieval dataset as the query class and utilize the image-to-query class similarity to re-rank the retrieval results. Extensive experiments are conducted on two publicly available RS image databases. Compared with the state-of-the-art methods, the proposed method can significantly enhance the retrieval precision.

Research on Image Retrieval Algorithm Based on Combination of Color and Shape Features

With the development of content-based image retrieval technology, the retrieval efficiency of image retrieval technology is getting higher and higher. For different data images, image retrieval based on color features and shape features can be used to improve retrieval efficiency. However, when a single image feature is retrieved, its retrieval efficiency still cannot meet people’s needs. In this paper, we propose an image retrieval algorithm based on the combination of color and shape features. The cumulative histogram method is used to calculate the color features of the image, and 7 Hu invariant moments are calculated as shape features. The color and shape features are combined with certain weights, and the Euclidean distance is used as the similarity measure. Finally, the image is retrieved, and the related experiments are passed. By comparing with related experiments, the algorithm effectively improves the accuracy of image retrieval.

Deep learning for noninvasive classification of clustered horticultural crops – A case for banana fruit tiers

Practical classification of some horticultural crops such as banana tiers, lanzones and grapes come into clusters instead of individual classification. Unlike most of classification studies, clustered crops are rarely studied due to their complex physical structure. A noninvasive deep learning classification of clustered banana given only a single image feature has been developed as a pioneering deep learning study for clustered horticultural crops. In recent deep learning developments, mask region-based convolution neural networks, also known as Mask R-CNN, show unique applications in image recognition by detecting objects within an image while simultaneously generating segmentation masks. With Mask R-CNN, detection of the complex banana fruit within an image predicts the banana class while at the same time generating a mask separating the fruit from its background. A real dataset is used based on banana tiers and the developed model discriminates normal from abnormal tiers. Unlike the previous general machine learning study, which discriminates reject class from normal class with classification accuracy of 79%, our deep learning model obtained a better averaged accuracy of 92.5%. The previous average weighted accuracy of 94.2% also improved to 96.1% with only a single image feature instead of tedious multiple image and size features. With data augmentation, the model slightly improved into 93.8% accuracy on classifying reject class and 96.5% for overall accuracy. Having successfully implemented in banana tiers, this deep learning classification can also serve as basis for other clustered horticultural crops.