Unknown Pixels Research Articles

FACNN: fuzzy-based adaptive convolution neural network for classifying COVID-19 in noisy CXR images.

COVID-19 detection using chest X-rays (CXR) has evolved as a significant method for early diagnosis of the pandemic disease. Clinical trials and methods utilize X-ray images with computer and intelligent algorithms to improve detection and classification precision. This article thus proposes a fuzzy-based adaptive convolution neural network (FACNN) model to improve the detection precision by confining the false rates. The feature extraction process between the successive regions is validated using a fuzzy process that classifies labeled and unknown pixels. The membership functions are derived based on high precision features for detection and false rate suppression process. The convolution neural network process is responsible for increasing detection precision through recurrent training based on feature availability. This availability analysis is verified using fuzzy derivatives under local variances. Based on variance-reduced features, the appropriate regions with labeled and unknown features are used for normal or infected classification. Thus, the proposed FACNN improves accuracy, precision, and feature extraction by 14.36%, 8.74%, and 12.35%, respectively. This model reduces the false rate and extraction time by 10.35% and 10.66%, respectively.

Tensorial Evolutionary Optimization for Natural Image Matting

Natural image matting has garnered increasing attention in various computer vision applications. The matting problem aims to find the optimal foreground/background (F/B) color pair for each unknown pixel and thus obtain an alpha matte indicating the opacity of the foreground object. This problem is typically modeled as a large-scale pixel pair combinatorial optimization (PPCO) problem. Heuristic optimization is widely employed to tackle the PPCO problem owing to its gradient-free property and promising search ability. However, traditional heuristic methods often encode F/B solutions to a one-dimensional (1D) representation and then evolve the solutions in a 1D manner. This 1D representation destroys the intrinsic two-dimensional (2D) structure of images, where the significant spatial correlations among pixels are ignored. Moreover, the 1D representation also brings operation inefficiency. To address the above issues, this article develops a spatial-aware tensorial evolutionary image matting (TEIM) method. Specifically, the matting problem is modeled as a 2D Spatial-PPCO (S-PPCO) problem, and a global tensorial evolutionary optimizer is proposed to tackle the S-PPCO problem. The entire population is represented as a whole by a third-order tensor, in which individuals are classified into two types: F and B individuals for denoting the 2D F/B solutions, respectively. The evolution process, consisting of three tensorial evolutionary operators, is implemented based on pure tensor computation for efficiently seeking F/B solutions. The local spatial smoothness of images is also integrated into the evaluation process for obtaining a high-quality alpha matte. Experimental results compared with state-of-the-art methods validate the effectiveness of TEIM.

An automatic method for rice mapping in Taishan, China using Sentinel-1A Time-series images

ABSTRACT Quickly and accurately obtaining information on rice planting areas is crucial for maintaining food security and social stability. Based on the time series of Sentinel-1A images, this study proposes an automatic rice mapping method called the Dynamic Threshold Time-Weighted Dynamic Time Warping method (DT-TWDTW). This method consists of two parts. First, rice samples are extracted automatically by a dynamic threshold method based on the unique flooding signal before and after rice transplanting. Then, the Time-Weighted Dynamic Time Warping (TWDTW) method is used for rice mapping based on the cumulative cost matrix between unknown pixels and rice samples. The results show that the DT-TWDTW method can achieve satisfactory mapping accuracy with an overall accuracy of 94.6%, higher than random forest (89.9%), and support vector machine (91.7%). This study provides a new approach and perspective for remote sensing monitoring of rice, especially for some regions without prior phenological knowledge and collection of the rice samples.

Quality Inpainting and Region Filling Using a Complete-Composite Patch Sampling Scheme (CCPSS) Based on Statistical Estimation

Research in the field of Inpainting and Region filling has nowadays gained immense momentum to account for historical and ancient mural conservation and the technological advancements as regards the need for image processing applications. This work proposes a complete Composite Patch Sampling Scheme (CCPSS) incorporating four refinements over Criminisi and other state-of-the-art techniques. The earlier Criminisi patch matching approach is supported by adding alien patches generated from independent color components of the existing known or valid patches. The dependency of confidence and data terms used in the priority function had been removed by selecting unknown pixels based on their neighborhood unknowns. The proceeding estimated unknown pixel patches are considered in estimating successive invalid pixel patches. Lastly, the highest similar or consistent known patch concerning an unknown patch is not finalized but retained until all the unknown patches are estimated. The true value of any unknown pixel is computed using the mean and median of all retained values. The inpainting, region filling, and reconstruction quality of the CCPSS scheme is superior in terms of visual perception when tested over high-resolution texture-structure images at the cost of computational overhead.

Dual-energy X-ray transmission identification method for coal and gangue with different thicknesses and densities

To reduce the thickness effect influence of dual-energy X-ray transmission (DE-XRT) and solve the problem of identifying coal and gangue with different thicknesses and densities, this paper proposes a method based on the Bayesian optimisation KNN model to classify the pixels of images. The gray feature I and R-values are used to train the classifier to recognize unknown classification pixels. The proportion of classified pixels in the images of a single target is used as the classification parameter. This method reduces the effect of thickness in R-I two-dimensional space. Considering the change in the density of different coal types and taking the density at the boundary between the coal and gangue as 1.8 g/cm3, the recognition rate reached 97.74 %. For raw coal with a 5–150 mm thickness range, the accuracy of identifying predischarged gangue reached 99.16 %. Our study provides a new method and practical guidance for DE-XRT coal preparation.

A Modified Version of the Direct Sampling Method for Filling Gaps in Landsat 7 and Sentinel 2 Satellite Imagery in the Coastal Area of Rhone River

Earth Observation (EO) data, such as Landsat 7 (L7) and Sentinel 2 (S2) imagery, are often used to monitor the state of natural resources all over the world. However, this type of data tends to suffer from high cloud cover percentages during rainfall/snow seasons. This has led researchers to focus on developing algorithms for filling gaps in optical satellite imagery. The present work proposes two modifications to an existing gap-filling approach known as the Direct Sampling (DS) method. These modifications refer to ensuring the algorithm starts filling unknown pixels (UPs) that have a specified minimum number of known neighbors (Nx) and to reducing the search area to pixels that share similar reflectance as the Nx of the selected UP. Experiments were performed on images acquired from coastal water bodies in France. The validation of the modified gap-filling approach was performed by imposing artificial gaps on originally gap-free images and comparing the simulated images with the real ones. Results indicate that satisfactory performance can be achieved for most spectral bands. Moreover, it appears that the bi-layer (BL) version of the algorithm tends to outperform the uni-layer (UL) version in terms of overall accuracy. For instance, in the case of B04 of an L7 image with a cloud percentage of 27.26%, accuracy values for UL and BL simulations are, respectively, 64.05 and 79.61%. Furthermore, it has been confirmed that the introduced modifications have indeed helped in improving the overall accuracy and in reducing the processing time. As a matter of fact, the implementation of a conditional filling path (minNx = 4) and a targeted search (n2 = 200) when filling cloud gaps in L7 imagery has contributed to an average increase in accuracy of around 35.06% and an average gain in processing time by around 78.18%, respectively.

GOSS: towards generalized open-set semantic segmentation

In this paper, we extend Open-set Semantic Segmentation (OSS) into a new image segmentation task called Generalized Open-set Semantic Segmentation (GOSS). Previously, with well-known OSS, the intelligent agents only detect unknown regions without further processing, limiting their perception capacity of the environment. It stands to reason that further analysis of the detected unknown pixels would be beneficial for agents’ decision-making. Therefore, we propose GOSS, which holistically unifies the abilities of two well-defined segmentation tasks, i.e. OSS and generic segmentation. Specifically, GOSS classifies pixels as belonging to known classes, and clusters (or groups) of pixels of unknown class are labelled as such. We propose a metric that balances the pixel classification and clustering aspects to evaluate this newly expanded task. Moreover, we build benchmark tests on existing datasets and propose neural architectures as baselines. Our experiments on multiple benchmarks demonstrate the effectiveness of our baselines. Code is made available at https://github.com/JHome1/GOSS_Segmentor.

Examining effect of super-resolution on AVIRIS-NG data: A precursor to generation of large-scale urban material and natural cover maps

The two-phase Airborne Visible InfraRed Imaging Spectrometer – Next Generation (AVIRIS-NG) campaign over India has generated datasets of numerous natural and artificial surroundings with high signal-to-noise ratios and higher spatial and spectral consistencies. However, the first phase AVIRIS-NG datasets of urban areas have 8.1 m spatial resolution, which is insufficient for producing level-4 urban material and natural cover maps required to model Earth’s processes at the micro-scale or constantly monitor urban processes. The current work addresses this drawback by demonstrating a part of the proposed proof-of-concept approach on the AVIRIS-NG dataset of Ahmedabad, India, wherein super-resolution (SR) is proposed to enhance the hyperspectral image’s spatial resolution and preserve its rich spectral content at the targeted spatial scale before utilizing it for producing detailed urban land cover maps. A step-wise account of the SR process has been provided herein to aid the reader in reproducing these experiments. Super-resolved products generated using four SR algorithms have been compared and validated using a composite framework based on processing time, qualitative visual investigation of selected image patches and spectral profiles, re-interpretation of Wald’s reduced resolution protocol and full-reference image quality metrics. Results suggest the AVIRIS-NG dataset’s successful spectral and spatial fidelity at a larger spatial scale, albeit varying in each super-resolved output. Sparse regression and natural prior (SRP), and anchored neighbourhood regression (ANR) produce the best and poorest super-resolved outputs among the tested SR algorithms. Although gradient profile prior (GPP) generated super-resolved product is the second best, it is also the most computationally expensive algorithm. An additional yardstick in the form of a spectral similarity analysis of randomly selected unknown spectral signatures on super-resolved output against a reference spectral library has been proposed to ascertain spectral content preservation further before utilizing the super-resolved product for urban information extraction. This exercise’s findings reveal the presence of the target urban land cover feature at the unknown pixels with matching scores of greater than 0.5 in both best and worst super-resolved products, thereby advocating the extension of the proposed proof-of-concept to recently launched spaceborne hyperspectral missions’ datasets.

Pose Mask: A Model-Based Augmentation Method for 2D Pose Estimation in Classroom Scenes Using Surveillance Images.

Solid developments have been seen in deep-learning-based pose estimation, but few works have explored performance in dense crowds, such as a classroom scene; furthermore, no specific knowledge is considered in the design of image augmentation for pose estimation. A masked autoencoder was shown to have a non-negligible capability in image reconstruction, where the masking mechanism that randomly drops patches forces the model to build unknown pixels from known pixels. Inspired by this self-supervised learning method, where the restoration of the feature loss induced by the mask is consistent with tackling the occlusion problem in classroom scenarios, we discovered that the transfer performance of the pre-trained weights could be used as a model-based augmentation to overcome the intractable occlusion in classroom pose estimation. In this study, we proposed a top-down pose estimation method that utilized the natural reconstruction capability of missing information of the MAE as an effective occluded image augmentation in a pose estimation task. The difference with the original MAE was that instead of using a 75% random mask ratio, we regarded the keypoint distribution probabilistic heatmap as a reference for masking, which we named Pose Mask. To test the performance of our method in heavily occluded classroom scenes, we collected a new dataset for pose estimation in classroom scenes named Class Pose and conducted many experiments, the results of which showed promising performance.

Crypt-OR: A privacy-preserving distributed cloud computing framework for object-removal in the encrypted images

Image inpainting is a technique to modify an image by removing/fill-up the undesired region(s) in a visually plausible manner. With the advancement of cloud applications, the cloud service providers (CSPs) provide image modifying services such as inpainting and undesired object removal to their users. However, these inpainting services require the user’s original image containing sensitive information that an adversary can misuse. In this paper, we address these privacy concerns in image enhancement techniques by proposing a novel privacy-preserving distributed-cloud-based framework for object removal in the encrypted images, namely Crypt-OR. The unknown pixels intensity value, obtained by removing the specified object(S), is approximated by incorporating the merits of the exemplar-based search–copy–paste approach and diffusion equation on the Shamir’s secret shares. The qualitative and quantitative analysis of Crypt-OR is evaluated over different real-world images with varying shaped and sized objects in the encrypted domain (ED). Crypt-OR outperforms the traditional exemplar-based object-removal schemes and is comparable with generative network-based inpainting schemes in the plain domain (PD). Further, Crypt-OR is proved information-theoretically secure in probabilistic and entropy viewpoints with standard cryptographic adversary attacks.

Progressive feature generation with mask awareness for image inpainting

Most existing image inpainting methods have achieved remarkable progress in repairing small image defects. However, when the holes are large, their filling contents suffer from structural distortion and center blur due to the weak correlation between known and unknown pixels. In this paper, we propose a Progressive Feature Generation (PFG) network which is mask awareable during the process of filling irregular holes. Specifically, in order to strengthen the constraint for the hole center, we propose dynamic partial convolution which can adaptively adjust the inpainting proportion according to mask ratio in each recurrence. To synthesize high-quality features in the feature generation phase, two parallel encoders are designed which could effectively improve the capability of model to restore large holes. We argue that during feature merging, the signals generated earlier at the same location are more credible. To this end, we develop the sub-regional weighted merging(SWM) method for PFG-Net to accurately merge the feature maps generated during the whole reconstruction process. Experiments on CelebA, Places2, and Paris StreetView datasets show that our model has excellent performance especially for large holes.

Efficient 2D image Upscaling using Iterative Optimized Sharpening filtering

In real-time image interpolation, polynomial-based approaches are used because they are simple and practical. The unknown pixels in the image grid are obtained using polynomial-based upscaling by taking into account the weighted average of adjacent pixels. It causes degradation in high frequency (HF) parts of the image due to blurring artifacts. Numerous edge-directed and learning-based techniques are discussed in the literature, which also introduces blurring in the image’s high variance region. To address the aforementioned issue, one pre-processing method based on the concept of unsharp masking is proposed in order to generate sharpened high resolution (HR) images from low resolution (LR) images. The LR image is blurred using a weighted average filter, which is based on the concept of unsharp masking. The difference between the LR image and the blurred LR image is then extracted to represent the loss edge portion of the image. This error or degraded HF image is sharpened iteratively using a cuckoo optimized sharpening filter known as the Iteratively Optimized Sharpening (IOS) filter. The subtle is then collected by iteratively applying ISO and combining it with the LR image using one optimized gain factor, resulting in a sharpened LR image. This pre-processing is done prior to interpolation to compensate for the loss of HF details. To up-sample the sharpened LR image, the MAKIMA scheme is used. Because of MAKIMA, the image’s edges and boundaries are preserved. As a result, an HR image with sharpened edges and texture details can be obtained. In comparison to existing techniques, the proposed algorithm yields better results in terms of visual quality and objectivity. Various image databases are used to assess the efficiency of the proposed method.

Development of a Phenology-Based Method for Identifying Sugarcane Plantation Areas in China Using High-Resolution Satellite Datasets

Sugarcane is an important sugar and biofuel crop with high socio-economic importance, and its planted area has increased rapidly in recent years. China is the world’s third or fourth sugarcane producer. However, to our knowledge, no study has investigated the mapping of sugarcane cultivation areas across entire China. In this study, we developed a phenology-based method to identify sugarcane plantations in China at 30-m spatial resolution from 2016–2020 using the time-series of Landsat and Sentinel-1/2 images derived from Google Earth Engine (GEE) platform. The method worked by comparing the phenological similarity in normalized difference vegetation index (NDVI) series between unknown pixels and sugarcane samples. The phenological similarity was assessed using the time-weighted dynamic time warping method (TWDTW), which has less sensitivity to training samples than machine learning methods and therefore can be easily applied to large areas with limited samples. More importantly, our method introduced multiple and moving time standard phenological curves of sugarcane to the TWDTW by fully considering the variable crop life-cycle of sugarcane, particularly its long harvest season spanning from December to March of the following year. Validations showed the method performed well in 2019, with overall accuracies of 93.47% and 92.74% for surface reflectance (SR) and top of atmosphere reflectance (TOA) data, respectively. The sugarcane maps agreed well with the agricultural statistical areas from 2016–2020. The mapping accuracies using TOA data were comparable to SR data in 2019–2020, but outperformed SR data in 2016–2018 when SR data had lower availability on GEE. The sugarcane maps produced in this study can be used to monitor growing conditions and production of sugarcane and, therefore, can benefit sugarcane management, sustainable sugarcane production, and national food security.

Generic Multispectral Demosaicking Based on Directional Interpolation

The low-cost snapshot multispectral spectral imaging systems with multispectral filter array (MSFA) require generic MSFA demosaicking methods to generate the multispectral image (MSI) with the variable number of spectral bands depending on the applications. Most of the existing MSFA demosaicking methods are either non-generic or perform inadequately. This paper presents a new generic MSFA demosaicking method based on the directional weighted interpolation. Our proposed method calculates the four directional estimates around the location of the unknown pixel and combines them in a weighted manner using the local edge magnitude in the corresponding directions to estimate the missing pixel values. Experimental results confirm that the proposed demosaicking method provides improvement, compared to the state-of-the-art generic demosaicking methods in terms of both subjective and objective evaluations on the two benchmark MSI datasets.

High Resolution Distribution Dataset of Double-Season Paddy Rice in China

Although China is the largest producer of rice, accounting for about 25% of global production, there are no high-resolution maps of paddy rice covering the entire country. Using time-weighted dynamic time warping (TWDTW), this study developed a pixel- and phenology-based method to identify planting areas of double-season paddy rice in China, by comparing temporal variations of synthetic aperture radar (SAR) signals of unknown pixels to those of known double-season paddy rice fields. We conducted a comprehensive evaluation of the method’s performance at pixel and regional scales. Based on 145,210 field surveyed samples from 2018 to 2020, the producer’s and user’s accuracy are 88.49% and 87.02%, respectively. Compared to county-level statistical data from 2016 to 2019, the relative mean absolute errors are 34.11%. This study produced distribution maps of double-season rice at 10 m spatial resolution from 2016 to 2020 over nine provinces in South China, which account for more than 99% of the planting areas of double-season paddy rice of China. The maps are expected to contribute to timely monitoring and evaluating rice growth and yield.

Mixed reality using illumination-aware gradient mixing in surgical telepresence: enhanced multi-layer visualization

Surgical telepresence using augmented perception has been applied, but mixed reality is still being researched and is only theoretical. The aim of this work is to propose a solution to improve the visualization in the final merged video by producing globally consistent videos when the intensity of illumination in the input source and target video varies. The proposed system uses an enhanced multi-layer visualization with illumination-aware gradient mixing using Illumination Aware Video Composition algorithm. Particle Swarm Optimization Algorithm is used to find the best sample pair from foreground and background region and image pixel correlation to estimate the alpha matte. Particle Swarm Optimization algorithm helps to get the original colour and depth of the unknown pixel in the unknown region. Our results showed improved accuracy caused by reducing the Mean squared Error for selecting the best sample pair for unknown region in 10 each sample for bowel, jaw and breast. The amount of this reduction is 16.48% from the state of art system. As a result, the visibility accuracy is improved from 89.4 to 97.7% which helped to clear the hand vision even in the difference of light. Illumination effect and alpha pixel correlation improves the visualization accuracy and produces a globally consistent composition results and maintains the temporal coherency when compositing two videos with high and inverse illumination effect. In addition, this paper provides a solution for selecting the best sampling pair for the unknown region to obtain the original colour and depth.

Determining Temporal Uncertainty of a Global Inland Surface Water Time Series

Earth observation time series are well suited to monitor global surface dynamics. However, data products that are aimed at assessing large-area dynamics with a high temporal resolution often face various error sources (e.g., retrieval errors, sampling errors) in their acquisition chain. Addressing uncertainties in a spatiotemporal consistent manner is challenging, as extensive high-quality validation data is typically scarce. Here we propose a new method that utilizes time series inherent information to assess the temporal interpolation uncertainty of time series datasets. For this, we utilized data from the DLR-DFD Global WaterPack (GWP), which provides daily information on global inland surface water. As the time series is primarily based on optical MODIS (Moderate Resolution Imaging Spectroradiometer) images, the requirement of data gap interpolation due to clouds constitutes the main uncertainty source of the product. With a focus on different temporal and spatial characteristics of surface water dynamics, seven auxiliary layers were derived. Each layer provides probability and reliability estimates regarding water observations at pixel-level. This enables the quantification of uncertainty corresponding to the full spatiotemporal range of the product. Furthermore, the ability of temporal layers to approximate unknown pixel states was evaluated for stratified artificial gaps, which were introduced into the original time series of four climatologic diverse test regions. Results show that uncertainty is quantified accurately (>90%), consequently enhancing the product’s quality with respect to its use for modeling and the geoscientific community.

Noise-Robust Image Reconstruction Based on Minimizing Extended Class of Power-Divergence Measures.

The problem of tomographic image reconstruction can be reduced to an optimization problem of finding unknown pixel values subject to minimizing the difference between the measured and forward projections. Iterative image reconstruction algorithms provide significant improvements over transform methods in computed tomography. In this paper, we present an extended class of power-divergence measures (PDMs), which includes a large set of distance and relative entropy measures, and propose an iterative reconstruction algorithm based on the extended PDM (EPDM) as an objective function for the optimization strategy. For this purpose, we introduce a system of nonlinear differential equations whose Lyapunov function is equivalent to the EPDM. Then, we derive an iterative formula by multiplicative discretization of the continuous-time system. Since the parameterized EPDM family includes the Kullback–Leibler divergence, the resulting iterative algorithm is a natural extension of the maximum-likelihood expectation-maximization (MLEM) method. We conducted image reconstruction experiments using noisy projection data and found that the proposed algorithm outperformed MLEM and could reconstruct high-quality images that were robust to measured noise by properly selecting parameters.

Fully convolutional open set segmentation

In traditional semantic segmentation, knowing about all existing classes is essential to yield effective results with the majority of existing approaches. However, these methods trained in a Closed Set of classes fail when new classes are found in the test phase, not being able to recognize that an unseen class has been fed. This means that they are not suitable for Open Set scenarios, which are very common in real-world computer vision and remote sensing applications. In this paper, we discuss the limitations of Closed Set segmentation and propose two fully convolutional approaches to effectively address Open Set semantic segmentation: OpenFCN and OpenPCS. OpenFCN is based on the well-known OpenMax algorithm, configuring a new application of this approach in segmentation settings. OpenPCS is a fully novel approach based on feature-space from DNN activations that serve as features for computing PCA and multi-variate gaussian likelihood in a lower dimensional space. In addition to OpenPCS and aiming to reduce the RAM memory requirements of the methodology, we also propose a slight variation of the method (OpenIPCS) that uses an iteractive version of PCA able to be trained in small batches. Experiments were conducted on the well-known ISPRS Vaihingen/Potsdam and the 2018 IEEE GRSS Data Fusion Challenge datasets. OpenFCN showed little-to-no improvement when compared to the simpler and much more time efficient SoftMax thresholding, while being some orders of magnitude slower. OpenPCS achieved promising results in almost all experiments by overcoming both OpenFCN and SoftMax thresholding. OpenPCS is also a reasonable compromise between the runtime performances of the extremely fast SoftMax thresholding and the extremely slow OpenFCN, being able to run close to real-time. Experiments also indicate that OpenPCS is effective, robust and suitable for Open Set segmentation, being able to improve the recognition of unknown class pixels without reducing the accuracy on the known class pixels. We also tested the scenario of hiding multiple known classes to simulate multimodal unknowns, resulting in an even larger gap between OpenPCS/OpenIPCS and both SoftMax thresholding and OpenFCN, implying that gaussian modeling is more robust to settings with greater openness.

A Novel Image Inpainting Framework Using Regression

In this article, a blockwise regression-based image inpainting framework is proposed. The core idea is to fill the unknown region in two stages: Extrapolate the edges to the unknown region and then fill the unknown pixels values in each sub-region demarcated by the extended edges. Canny edge detection and linear edge extension are used to respectively identify and extend edges to the unknown region followed by regression within each sub-region to predict the unknown pixel values. Two different regression models based on K-nearest neighbours and support vectors machine are used to predict the unknown pixel values. The proposed framework has the advantage of inpainting without requiring prior training on any image dataset. The extensive experiments on different images with contrasting distortions demonstrate the robustness of the proposed framework and a detailed comparative analysis shows that the proposed technique outperforms existing state-of-the-art image inpainting methods. Finally, the proposed techniques are applied to MRI images suffering from susceptibility artifacts to illustrate the practical usage of the proposed work.