Neighboring Pixels Research Articles

Rain detection and removal from slowly varying background videos

Rain in bad weather becomes very challenging issue for automated driver assistance systems, security applications and many other computer vision algorithms. Number of rain removal algorithms have been proposed in these years but none of them has been able to increase the accuracy to large extent. In this paper a novel algorithm has been proposed where the slowly varying back ground scenes are taken for analysis. One part of the algorithm is based on the detection of rain pixels and the other part is removing them such that the original image must not lose its originality. Initially one background is modelled by taking the intensities of neighbourhoods of pixels. Then rain is detected by comparing each pixel with the modelled back ground followed by updating the background. The detected rain pixels are repaired with the help of some temporal neighbours. The results are promising to prove the work.

Adaptive Mask Sampling and Manifold to Euclidean Subspace Learning With Distance Covariance Representation for Hyperspectral Image Classification

For the abundant spectral and spatial information recorded in hyperspectral images (HSIs), fully exploring spectral-spatial relationships has attracted widespread attention in hyperspectral image classification (HSIC) community. However, there are still some intractable obstructs. For one thing, in the patch based processing pattern, some spatial neighbor pixels are often inconsistent with the central pixel in land-cover class. For another thing, linear and nonlinear correlations between different spectral bands are vital yet tough for representing and excavating. To overcome these mentioned issues, an adaptive mask sampling and manifold to Euclidean subspace learning (AMS-M2ESL) framework is proposed for HSIC. Specifically, an adaptive mask based intra-patch sampling (AMIPS) module is firstly formulated for intra-patch sampling in an adaptive mask manner based on central spectral vector oriented spatial relationships. Then, based on distance covariance descriptor, a dual channel distance covariance representation (DC-DCR) module is proposed for modeling unified spectral-spatial feature representations and exploring spectral-spatial relationships, especially linear and nonlinear interdependence in spectral domain. Furthermore, considering that distance covariance matrix lies on the symmetric positive definite (SPD) manifold, we implement a manifold to Euclidean subspace learning (M2ESL) module respecting Riemannian geometry of SPD manifold for high-level spectral-spatial feature learning. Additionally, we introduce an approximate matrix square-root (ASQRT) layer for efficient Euclidean subspace projection. Extensive experimental results on three popular HSI data sets with limited training samples demonstrate the superior performance of the proposed method compared with other state-of-the-art methods. The source code is available at https://github.com/lms-07/AMS-M2ESL.

A Fast Spatial–Spectral NMF for Hyperspectral Unmixing

This letter proposes a fast yet efficient method to solve the hyperspectral unmixing problem in the challenging unsupervised context, i.e., when the endmember spectral signatures are unknown. First, a coarse approximation of the hyperspectral image is computed by spatially averaging neighboring pixels, which significantly reduces the amount of pixels to be handled. This reduced set of hyperspectral pixels is unmixed to derive coarse solutions of the unmixing problem, i.e., coarse estimates of the endmember signatures and the corresponding low-resolution abundance maps. Then, the plain resolution abundance maps are estimated from the corresponding hyperspectral image based on the coarse endmember signatures. A sparsity promoting prior exploiting the low resolution map complements the conventional data fitting term to promote spatial smoothness while mitigating the loss of details in the edge areas. Finally, a least square optimization problem is solved to obtain the actual endmember signatures from the hyperspectral image and the abundance maps of plain resolution estimated in the previous step. Numerical experiments show that the proposed method is fast and performs well compared to state-of-the-art approaches from the literature.

Graph Meta Transfer Network for Heterogeneous Few-Shot Hyperspectral Image Classification

Since obtaining labeled hyperspectral images (HSIs) is difficult and time-consuming, the shortage of training samples has always been a challenge for HSI classification. In practical applications, only a few labeled samples are available in the task domain (target domain), while sufficient labeled samples are available in another domain (source domain). At the same time, these two domains are heterogeneous and contain different categories. This scenario makes it difficult to effectively transfer knowledge from the source domain to the target domain. To address this challenge, we propose a novel heterogeneous few-shot learning (FSL) method, namely graph meta transfer network (GMTN). Specifically, the graph sample and aggregate network (GraphSAGE) and meta-learning, which are both inductive learning, are integrated into a unified framework. In this way, the aggregation function is generalized from abundant few-shot tasks for feature extraction on the source and target domains. The spatial importance strategy (SIS) is designed to guide the feature propagation and alleviate the information interference caused by different categories. The neighborhood receptive field spectral attention (RFSA) mechanism is proposed to model the importance of spectral band using the information of the neighborhood pixels, which enables GMTN to pay more attention to bands with discriminative features in both domains. In addition, the node spatial information reset method is proposed to augment samples based on the spatial position relationship of nodes. Furthermore, to alleviate the domain shift in heterogeneous scenarios, the conditional domain adversarial strategy is used to achieve effective meta-knowledge transfer. Experiments show that GMTN outperforms the compared state-of-the-art methods.

Depth Estimation Method of Light Field Based on Attention Mechanism of Neighborhood Pixel

Depth Estimation Method of Light Field Based on Attention Mechanism of Neighborhood Pixel

Modeling and prediction of fire occurrences along an elevational gradient in Western Himalayas

Forest fires are the result of complex interactions among human, geographic and weather conditions. Climate change would alter the link between forest fire and the controlling factors. The objective of the study is to model the forest fire occurrences and quantify the contribution of explanatory geographic, climatic and anthropogenic variables using satellite-derived historical fire data (2003–2019) and machine learning classifiers over the western Himalaya, India. The climatic variables were derived from a regional Earth system model (ROM). Along with the key selected explanatory variables, the conditions of neighbouring (3 × 3) pixels were incorporated to account for the contribution from the surrounding area. Out of the selected classifiers, random forest recorded the most promising performance in k-fold cross-validation (f2-score = 0.95 and f1-score = 0.94) as well as in the final model validation (f2-score = 0.85 and f1-score = 0.84). The elevation and mean neighbour elevation exhibited the highest influence (8.18% and 6.72%, respectively) in forest fire occurrences followed by near-surface temperatures (4.65–5.78%). We predicted the forest fire susceptibility [0, 1] for 2030, 2040 and 2050 using the future climate projections. The predicted map can be useful to plan effective fire management strategies to minimize damage to the forest ecosystem.

A Method for Remote Sensing Image Classification by Combining Pixel Neighbourhood Similarity and Optimal Feature Combination

In the study of remote sensing image classification, feature extraction and selection is an effective method to distinguish different classification targets. Constructing a high-quality spectral-spatial feature and feature combination has been a worthwhile topic for improving classification accuracy. In this context, this study constructed a spectral-spatial feature, namely the Pixel Neighbourhood Similarity (PNS) index. Meanwhile, the PNS index and 19 spectral, textural and terrain features were involved in the Correlation-based Feature Selection (CFS) algorithm for feature selection to generate a feature combination (PNS-CFS). To explore how PNS and PNS-CFS improve the classification accuracy of land types. The results show that: (1) The PNS index exhibited clear boundaries between different land types. The performance quality of PNS was relatively highest compared to other spectral-spatial features, namely the Vector Similarity (VS) index, the Change Vector Intensity (CVI) index and the Correlation (COR) index. (2) The Overall Accuracy (OA) of the PNS-CFS was 94.66% and 93.59% in study areas 1 and 2, respectively. These were 7.48% and 6.02% higher than the original image data (ORI) and 7.27% and 2.39% higher than the single-dimensional feature combination (SIN-CFS). Compared to the feature combinations of VS, CVI, and COR indices (VS-CFS, CVI-COM, COR-COM), PNS-CFS had the relatively highest performance and classification accuracy. The study demonstrated that the PNS index and PNS-CFS have a high potential for image classification.

LTF-NSI: a novel local transfer function based on neighborhood similarity index for medical image enhancement

Medical image is an essential tool used in quantitative and qualitative evaluation of different diseases. Medical imaging methods such as fluorescein angiography (FA), optical coherence tomography angiography (OCTA), computed tomography (CT), optical coherence tomography (OCT), and X-ray are used for diagnosis. These imaging modalities suffer from low contrast, which leads to deterioration in the image quality. Consequently, this causes limitation in the usage of medical images in clinical routine and hindered its potential by depriving clinicians from assessing useful information that are needed in disease monitoring, treatment, progression, and decision-making. To overcome this limitation, we propose a novel local transfer function for medical image enhancement algorithm using the pixel neighborhood constraint. The proposed algorithm uses block-wise intensity distribution to generate the regional similarity index. The regional similarity index transformed each centered pixel in the block, to generate a new similarity image. An intuitive optimization algorithm is utilized to optimize the proposed algorithm parameters. Experimentation results show that the proposed LTF-NSI performs better than the state-of-the-art methods and improves the interpretability and perception of the medical images, which can provide clinicians and computer vision program with good quantitative and qualitative information.

Three-Stage Tone Mapping Algorithm

In this paper, a tone mapping algorithm is presented to map real-world luminance into displayed luminance. Our purpose is to reveal the local contrast of real-world scenes on a conventional monitor. Around this point, we propose a three-stage algorithm to visualize high dynamic range images. All pixels of high dynamic range images are classified into three groups. For the first stage, we introduce piecewise linear mapping as the global tone mapping operator to map the luminance of the first group, which provides overall impressions of luminance. For the second stage, the luminance of the second group is determined by the weighted average of its neighborhood pixels, which are derived from the first group’s pixels. For the third stage, the luminance of the third group is determined by the weighted average of its neighborhood pixels, which are derived from the second group’s pixels. Experimental results on several real-world images and the TMQI database show that our algorithm can improve the visibility of real-world scenes with about 12% and 9% higher scores of mean opinion score and tone-mapped image quality index than the closest competitive tone mapping methods. Compared to the existing tone mapping methods, our algorithm produces visually compelling results without halo artifacts and loss of detail.

Height aware understanding of remote sensing images based on cross-task interaction

3D scene perception that provides semantic interpretation for each point has a wide range of practical scenarios. However, existing methods are generally implemented based on 3D point cloud. In this work, we investigate the inverse projection problem from 2D plane image to 3D scene perception and propose a multi-task model to jointly predict height information and semantic categories. Taking into account the commonalities and differences between the two tasks, we have carefully designed the multi-task model to explicitly strengthen the establishment of cross-task correlations. Specifically, the calibration refinement attention (CRA) module is proposed before the classifier heads, incorporating the beneficial information while filtering the inconsistent characteristics among the two tasks. Besides, a spatial structure enhanced (SSE) module is introduced to integrate the spatial structure information into the output features of CRA module through skip-connection. After that, the neighboring pixel affinity (NPA) loss and the soft weighted ordinal (SWO) classification loss for the two tasks are introduced to optimize the direction of the task gradients. At the same time, to validate the effectiveness of the proposed method, a novel metric named height constrained semantic accuracy (HCSA) is proposed to consider the accuracy of semantic segmentation and height estimation jointly. Extensive experiments on Vaihingen, Potsdam, and DFC2019 demonstrate that the proposed method achieves generalization and competitive performance.

Robust interval type-2 kernel-based possibilistic fuzzy clustering algorithm incorporating local and non-local information

Type-2 fuzzy set theory has certain potential advantage in processing high-order uncertainty, and has been more and more widely applied in pattern recognition, image processing, and system modeling. Although it has been introduced into robust fuzzy clustering modeling, most clustering methods based on type-2 fuzzy set theory have many obvious shortcomings, such as weak robustness against noise and high sensitivity to initial value. Therefore, this paper presents a novel single fuzzifier interval type-2 kernel-based possibilistic fuzzy local and non-local information c-means clustering, and it is driven by deep neighborhood information for image segmentation in the presence of high noise. Firstly, we construct the novel deep neighborhood window structure, which consists of local neighborhood window around current pixel and deep neighborhood window around the pixel in local neighborhood window around current pixel. Secondly, using the structural similarity between local neighborhood window and deep neighborhood window around current pixel, we construct a novel fuzzy local information factor to tune the impact of local neighborhood information on current pixel clustering. Thirdly, possibility theory and fuzzy local information factor are introduced into interval type-2 kernel-based fuzzy c-means clustering, and a novel single fuzzifier type-2 possibilistic fuzzy clustering with richer local and non-local information and kernel metric is proposed. Experimental results indicate that the proposed algorithm outperforms existing state-of-the-art robust fuzzy clustering-related algorithms, and significantly improve the robustness of robust fuzzy clustering algorithms against noise.

Reversible Data Hiding By Using CNN Prediction and Adaptive Embedding.

In the field of reversible data hiding (RDH), how to predict an image and embed a message into the image with smaller distortion are two important aspects. In this paper, we propose a novel and efficient RDH method by innovating an intelligent predictor and an adaptive embedding way. In the prediction stage, we first constructed a convolutional neural network (CNN) based predictor by reasonably dividing an image into four parts. In such a way, each part can be predicted by using the other three parts as the context for the improvement of the prediction performance. Compared with existing predictors, the proposed CNN predictor can use more neighboring pixels for the prediction by exploiting its multi-receptive fields and global optimization capacities. In the embedding stage, we also developed a prediction-error-ordering (PEO) based adaptive embedding strategy, which can better adapt image content and thus efficiently reduce the embedding distortion by elaborately and luminously applying background complexity to select and pair those smaller prediction errors for data hiding. With the proposed CNN prediction and embedding ways, the RDH method presented in this paper provides satisfactory results in improving the visual quality of data hidden images, e.g., the average PSNR value for the Kodak benchmark dataset can reach as high as 63.59 dB with an embedding capacity of 10,000 bits. Extensive experimental results have shown that the RDH method proposed in this paper is superior to those existing state-of-the-art works.

Laplacian of Logarithm as Illuminant-Invariant Input Space

An object’s color is affected by the color of the light incident upon it, and the illuminant-dependent nature of color creates problems for convolutional neural networks performing tasks such as image classification and object recognition. Such networks would benefit from illuminant-invariant representation of the image colors. The Laplacian of the logarithm of the image is introduced as an effective color invariant. Applying the Laplacian in log space makes the input colors approximately illuminationinvariant. The illumination invariance derives from the fact that finite-difference differentiation in log space is equivalent to ratios of neighboring pixels in the original space. For narrow-band sensors, rationing neighboring pixels cancels out their shared illumination component. The resulting color representation is no longer absolute, but rather is a relative color representation. Testing shows that when using the Laplacian of the logarithm as input to a Convolutional Neural Network designed for classification its performance is: (i) approximately equal to that of the same network trained on sRGB data under white light, and (ii) largely unaffected by changes in the illumination.

Dark Spot Detection from SAR Images Based on Superpixel Deeper Graph Convolutional Network

Synthetic Aperture Radar (SAR) is the primary equipment used to detect oil slicks on the ocean’s surface. On SAR images, oil spill regions, as well as other places impacted by atmospheric and oceanic phenomena such as rain cells, upwellings, and internal waves, appear as dark spots. Dark spot detection is typically the initial stage in the identification of oil spills. Because the identified dark spots are oil slick candidates, the quality of dark spot segmentation will eventually impact the accuracy of oil slick identification. Although certain sophisticated deep learning approaches employing pixels as primary processing units work well in remote sensing image semantic segmentation, finding some dark patches with weak boundaries and small regions from noisy SAR images remains a significant difficulty. In light of the foregoing, this paper proposes a dark spot detection method based on superpixels and deeper graph convolutional networks (SGDCNs), with superpixels serving as processing units. The contours of dark spots can be better detected after superpixel segmentation, and the noise in the SAR image can also be smoothed. Furthermore, features derived from superpixel regions are more robust than those derived from fixed pixel neighborhoods. Using the support vector machine recursive feature elimination (SVM-RFE) feature selection algorithm, we obtain an excellent subset of superpixel features for segmentation to reduce the learning task difficulty. After that, the SAR images are transformed into graphs with superpixels as nodes, which are fed into the deeper graph convolutional neural network for node classification. SGDCN leverages a differentiable aggregation function to aggregate the node and neighbor features to form more advanced features. To validate our method, we manually annotated six typical large-scale SAR images covering the Baltic Sea and constructed a dark spot detection dataset. The experimental results demonstrate that our proposed SGDCN is robust and effective compared with several competitive baselines. This dataset has been made publicly available along with this paper.

Exploring Intra- and Inter-Video Relation for Surgical Semantic Scene Segmentation.

Automatic surgical scene segmentation is fundamental for facilitating cognitive intelligence in the modern operating theatre. Previous works rely on conventional aggregation modules (e.g., dilated convolution, convolutional LSTM), which only make use of the local context. In this paper, we propose a novel framework STswinCL that explores the complementary intra- and inter-video relations to boost segmentation performance, by progressively capturing the global context. We firstly develop a hierarchy Transformer to capture intra-video relation that includes richer spatial and temporal cues from neighbor pixels and previous frames. A joint space-time window shift scheme is proposed to efficiently aggregate these two cues into each pixel embedding. Then, we explore inter-video relation via pixel-to-pixel contrastive learning, which well structures the global embedding space. A multi-source contrast training objective is developed to group the pixel embeddings across videos with the ground-truth guidance, which is crucial for learning the global property of the whole data. We extensively validate our approach on two public surgical video benchmarks, including EndoVis18 Challenge and CaDIS dataset. Experimental results demonstrate the promising performance of our method, which consistently exceeds previous state-of-the-art approaches. Code is available at https://github.com/YuemingJin/STswinCL.

Image Signal Decomposition Using Polynomial Representation with Hybrid Lossy and Non-Lossy Coding Scheme

This article presents a polynomial-based image compression scheme, which consists of using the color model (YUV) to represent color contents and using two-dimensional polynomial coding (first-order) with variable block size according to correlation between neighbor pixels. The residual part of the polynomial for all bands is analyzed into two parts, most important (big) part, and least important (small) parts. Due to the significant subjective importance of the big group; lossless compression (based on Run-Length spatial coding) is used to represent it. Furthermore, a lossy compression system scheme is utilized to approximately represent the small group; it is based on an error-limited adaptive coding system and using the transform coding scheme (discrete cosine transform or bi-orthogonal transform). Experimentally, the developed system has achieved high compression ratios with acceptable quality for color images. The performance results are comparable to those introduced in recent studies; the accomplishment of the introduced image compression system was analyzed and compared with the performance of the JPEG standard. The results of the developed system show better performance than that of the JPEG standard.

Graph-Based Image Segmentation for Road Extraction from Post-Disaster Aerial Footage

This research effort proposes a novel method for identifying and extracting roads from aerial images taken after a disaster using graph-based image segmentation. The dataset that is used consists of images taken by an Unmanned Aerial Vehicle (UAV) at the University of West Florida (UWF) after hurricane Sally. Ground truth masks were created for these images, which divide the image pixels into three categories: road, non-road, and uncertain. A specific pre-processing step was implemented, which used Catmull–Rom cubic interpolation to resize the image. Moreover, the Gaussian filter used in Efficient Graph-Based Image Segmentation is replaced with a median filter, and the color space is converted from RGB to HSV. The Efficient Graph-Based Image Segmentation is further modified by (i) changing the Moore pixel neighborhood to the Von Neumann pixel neighborhood, (ii) introducing a new adaptive isoperimetric quotient threshold function, (iii) changing the distance function used to create the graph edges, and (iv) changing the sorting algorithm so that the algorithm can run more effectively. Finally, a simple function to automatically compute the k (scale) parameter is added. A new post-processing heuristic is proposed for road extraction, and the Intersection over Union evaluation metric is used to quantify the road extraction performance. The proposed method maintains high performance on all of the images in the dataset and achieves an Intersection over Union (IoU) score, which is significantly higher than the score of a similar road extraction technique using K-means clustering.

Abnormal behavior capture of video dynamic target based on 3D convolutional neural network.

The use of computers to understand video content can accurately and quickly label various videos. Behavior recognition technology can help users filter the video by screening the content. However, this calculation mode, which is only sensitive to the features in a pixel neighborhood, cannot effectively extract cross-frame long-range video features. In addition, the common long-range dependency capture methods are based on pixel pairs, which contain less semantic information and cannot accurately model dependencies. Based on this, this paper generates semantic units with rich semantic information in the form of neighborhood pixel aggregation and proposes a multi-semantic long-range dependency capture algorithm to solve this problem, which makes the established dependency relationship more accurate. At the same time, this paper proposes an early dependency transfer technology to speed up the reasoning speed of the multi-semantic long-range dependency capture algorithm. By embedding the proposed algorithm into the original convolutional neural network, and conducting sufficient performance tests and evaluations on different data sets, it is shown that the proposed algorithm outperforms other current algorithms in terms of recognition accuracy and achieves the optimal recognition effect, which can effectively enhance the long-range dependency capture ability and temporal modeling ability of the convolutional network, and improve the quality of video feature representation.

Semi-supervised label propagation for multi-source remote sensing image change detection

Remote sensing image change detection remains a challenging task. Most existing approaches are based on fully supervised learning, but labeled data are so scarce for change detection. It is difficult to exhibit high detection performance with a limited amount of labeled data. In this paper, we propose a semi-supervised Label Propagation (SSLP) approach for multi-source remote sensing image change detection. First, a clustering label propagation (CLP) method is designed to cluster pre and post images, respectively, and assign pseudo labels to unlabeled pixel pairs that have similar mapping relationships to labeled pixel pairs. Second, a pixel density metric is investigated to filter out the data with low density and retain the data with high density, which can ensure the reliability of the propagated data. Third, a secondary expansion method based on pixel neighborhood is used to generate enough training data for training a classifier. Finally, the effectiveness of SSLP is validated on three real datasets by comparing to other related methods.

Breast Ultrasound Image Segmentation Algorithm Using Adaptive Region Growing and Variation Level Sets

To address the features of strong noise, blurred boundaries, and poor imaging quality in breast ultrasound images, we propose a method for segmenting breast ultrasound images using adaptive region growing and variation level sets. First, this method builds a template layer from the difference between the marked image and the original image. Second, the Otsu algorithm is used to measure the target and background using the maximum class variance method to set the threshold. Finally, through the level set of the pixel neighborhood, the boundary points of the adaptive region growth are specified by the level set of the pixel neighborhood, and it is therefore possible to accurately determine the contour perimeter and area of the lesion region. The results demonstrate that the value of Jaccard and Dice for benign tumors is greater than 0.99. Therefore, the segmentation effect of breast images can be achieved by utilizing a breast ultrasound image segmentation approach that uses adaptive region growth and variation level sets.