Similar Pixels Research Articles

CUR Transformer: A Convolutional Unbiased Regional Transformer for Image Denoising

Image denoising is a fundamental problem in computer vision and multimedia computation. Non-local filters are effective for image denoising. But existing deep learning methods that use non-local computation structures are mostly designed for high-level tasks, and global self-attention is usually adopted. For the task of image denoising, they have high computational complexity and have a lot of redundant computation of uncorrelated pixels. To solve this problem and combine the marvelous advantages of non-local filter and deep learning, we propose a Convolutional Unbiased Regional (CUR) transformer. Based on the prior that, for each pixel, its similar pixels are usually spatially close, our insights are that (1) we partition the image into non-overlapped windows and perform regional self-attention to reduce the search range of each pixel, and (2) we encourage pixels across different windows to communicate with each other. Based on our insights, the CUR transformer is cascaded by a series of convolutional regional self-attention (CRSA) blocks with U-style short connections. In each CRSA block, we use convolutional layers to extract the query, key, and value features, namely Q , K , and V , of the input feature. Then, we partition the Q , K , and V features into local non-overlapped windows and perform regional self-attention within each window to obtain the output feature of this CRSA block. Among different CRSA blocks, we perform the unbiased window partition by changing the partition positions of the windows. Experimental results show that the CUR transformer outperforms the state-of-the-art methods significantly on four low-level vision tasks, including real and synthetic image denoising, JPEG compression artifact reduction, and low-light image enhancement.

A Sparse Capture Light-Field Coding Algorithm Based on Target Pixel Matching for a Multi-Projector-Type Light-Field Display System

The traditional light-field coding algorithm used in a multi-projector-type light-field display system requires sophisticated and complex three-dimensional modeling processes or parallax images obtained through dense capture. Here we propose an algorithm based on target pixel matching, which directly uses parallax images without a complex modeling process, and can achieve a more accurate light-field reconstruction effect under sparse capture conditions. For the lack of capture information caused by sparse capture, this algorithm compares the pixel similarity of the captured images of the same object point on different cameras to accurately determine the real capture information of the object point at different depths, which is recorded as the target pixel, and then the target pixel is encoded according to the lighting path to obtain the correct projector image array (PIA). By comparing the quality of PIAs generated by the traditional light-field coding algorithm and the display effect after loading the PIAs into the actual display system, we proved the effectiveness of the algorithm.

Two-Stage Cascaded CNN Model for 3D Mitochondria EM Segmentation

Mitochondria are the organelles that generate energy for the cells. Many studies have suggested that mitochondrial dysfunction or impairment may be related to cancer and other neurodegenerative disorders such as Alzheimer’s and Parkinson’s diseases. Therefore, morphologically detailed alterations in mitochondria and 3D reconstruction of mitochondria are highly demanded research problems in the performance of clinical diagnosis. Nevertheless, manual mitochondria segmentation over 3D electron microscopy volumes is not a trivial task. This study proposes a two-stage cascaded CNN architecture to achieve automated 3D mitochondria segmentation, combining the merits of top-down and bottom-up approaches. For top-down approaches, the segmentation is conducted on objects’ localization so that the delineations of objects’ contours can be more precise. However, the combinations of 2D segmentation from the top-down approaches are inadequate to perform proper 3D segmentation without the information on connectivity among frames. On the other hand, the bottom-up approach finds coherent groups of pixels and takes the information of 3D connectivity into account in segmentation to avoid the drawbacks of the 2D top-down approach. However, many small areas that share similar pixel properties with mitochondria become false positives due to insufficient information on objects’ localization. In the proposed method, the detection of mitochondria is carried out with multi-slice fusion in the first stage, forming the segmentation cues. Subsequently, the second stage is to perform 3D CNN segmentation that learns the pixel properties and the information of 3D connectivity under the supervision of cues from the detection stage. Experimental results show that the proposed structure alleviates the problems in both the top-down and bottom-up approaches, which significantly accomplishes better performance in segmentation and expedites clinical analysis.

Classification of tea plantation using orthomosaics stitching maps from aerial images based on CNN

In Indonesia, Tea is an important economic crop that is widely grown, and in many countries, accurate mapping of tea plantations is essential for the operation, management, and monitoring of the growth and development of the tea industry. We propose a classification of tea plantations using orthomosaics from aerial images based on the Convolutional Neural Network (CNN) which identifies the condition of the tea plantations with the parameters observed, namely the condition of the tea leaves, estimated yields achieved, and monitoring of treeless areas caused by tree death. In this study, we took a sample of 20 hectares. We classify images based on maps generated by drones in previous studies. Image segmentation is performed to maintain image objects, while an enhanced CNN model is used to extract deep image features. To get complete results, this study uses UAV (Unmanned Aerial Vehicle) imagery as the basis for the map, which is then combined or stacked into one image. The results of the images that are used as maps undergo image classification, where the information contained in the map is mapped and divided according to its type. The area of ​​the tea plantations sampled is 20 ha, and the threshold for the image captured by the UAV is 5% of the total area captured, which is around 1 ha. If the image created by the UAV has an error of more than 5%, then the image does not meet the classification requirements. We determine this margin of error based on the performance of the drone camera capture when capturing Fig. 2, and the resolution used is 4096 x 2160 for each image captured by the drone. We conclude that the proposed method for mapping tea plantations using ultra-high resolution remote sensing imagery is effective and has great potential for mapping tea plantations in areas such as the development of drone aerial photography methods for tea plantations based on image classification for forecasting. tea plantations Image stitching can be used to improve the monitoring of tea plantations and predict harvest time using a classification process. The tea garden map has 5 types of information categorized by harvest time, medium leaf tea, milled tea, tea, and old tea. The success of image recognition shows the error matrix data by testing 123 random points spread over the map, of which 113 random points were identified with an average accuracy of 91.87%, this value is of course very good and exceeds the specified threshold of 75%. When using this method, an error occurs that the colors of similar pixels cannot be distinguished, resulting in an incorrect detection. In addition, the image stitching method using the orthomosaics method has succeeded in performing image stitching and can be well applied to classification using the CNN approach.

Virtual image-based cloud removal for Landsat images

ABSTRACT The inevitable thick cloud contamination in Landsat images has severely limited the usability and applications of these images. Developing cloud removal algorithms has been a hot research topic in recent years. Many previous algorithms used one or multiple cloud-free image(s) in the same area acquired on other date(s) as reference image(s) to reconstruct missing pixel values. However, it remains challenging to determine the optimal reference image(s). In addition, abrupt land cover change can substantially degrade the reconstruction accuracies. To address these issues, we present a new cloud removal algorithm called Virtual Image-based Cloud Removal (VICR). For each cloud region, VICR reconstructs the missing surface reflectance by three steps: virtual image within cloud region construction based on time-series reference images, similar pixel selection using the newly proposed temporally weighted spectral distance (TWSD), and residual image estimation. By establishing two buffer zones around the cloud region, VICR allows automatic selection of the optimal set of time-series reference images. The effectiveness of VICR was validated at four testing sites with different landscapes (i.e. urban, croplands, and wetlands) and land change patterns (i.e. phenological change, abrupt change caused by flooding and tidal inundation), and the performances were compared with mNSPI (modified neighborhood similar pixel interpolator), WLR (weighted linear regression) and ARRC (AutoRegression to Remove Clouds). Experimental results showed that VICR outperformed the other algorithms and achieved higher Correlation Coefficients and lower Root Mean Square Errors in surface reflectance estimation at the four sites. The improvement is particularly noticeable at the sites with abrupt land change. By considering the difference in the contributions from the reference images, TWSD can select more reliable similar pixels to improve the prediction of abrupt change in surface reflectance. Moreover, VICR is more robust to different cloud sizes and to changing reference images. VICR is also computationally much faster than ARRC. The framework for time-series image cloud removal by VICR has great potential to be applied for large datasets processing.

Depth and Pixel-Distance Based Attention for Outdoor Semantic Segmentation

Semantic segmentation has been a crucial technology for various practical applications such as autonomous driving. Recently, attempts have been made to improve the performance of semantic segmentation using depth information. However, most attempts have been focused on the indoor environment for the following reasons. First, it is relatively more difficult to obtain accurate and dense depth information outdoors. Second, a network with a new structure is required to use depth information because processing depth as an input demands an additional encoder. To overcome aforementioned difficulties, we propose a novel Depth and Pixel-distance based Attention (DPA) module, which utilizes depth information to compute the similarity between pixels. The similarity of pixels is computed using the fact that pixels belonging to the same object have similar depth values. Because only the relative difference in depth is considered, it is relatively robust despite the accuracy of the provided depth information. Furthermore, DPA is a simple plug-in module that can be applied to existing RGB-based segmentation backbones. Since no encoder is added, it is much more efficient in terms of computation. We conduct extensive experiments on the Cityscapes dataset using various baseline architectures. Regardless of the baseline models, DPA yields meaningful performance improvements in semantic segmentation tasks. It is also computationally more efficient compared to the methods that take depth information as input.

Soft Instance-Level Domain Adaptation With Virtual Classifier for Unsupervised Hyperspectral Image Classification

Adversarial learning-based unsupervised hyperspectral image (HSI) classification methods usually adapt probability distributions by minimizing the statistical distance between similar pixels of different HSIs. Since the adversarial learning may weaken the discriminability of features, the extracted features will contain a lot of non-discriminative information, pixels with similar features may be classified as different classes. Therefore, directly reducing the statistical distance between similar pixels in a latent space may aggravate misclassification. To this end, we propose an unsupervised HSI classification method called soft instance-level domain adaptation with virtual classifier. First, the domain-invariant features of HSI are extracted by a graph convolutional network. Then, a feature similarity metric-based virtual classifier is constructed to output class probabilities of target-domain samples. Furthermore, to enable similar features of HSIs from different domains to be classified into the same class, the divergence between the real and virtual classifiers is reduced by minimizing the real and virtual classifier determinacy disparity. Finally, to reduce the influence of noisy pseudo-labels, a soft instance-level domain adaptation method is proposed. For each target-domain sample, the confidence coefficients are assigned to its corresponding positive and negative samples in the source domain, and a soft prototype contrastive loss is constructed and minimized to adapt two domains in an instance-level way. Experimental results on five real HSI datasets including Botswana, Kennedy Space Center, Pavia Center, Pavia University, and HyRANK demonstrate the effectiveness of our proposed method.

Classification via Structure-Preserved Hypergraph Convolution Network for Hyperspectral Image

Graph convolutional network (GCN) as a combination of deep learning and graph learning has gained increasing attention in hyperspectral image (HSI) classification. However, most GCN methods consider the simple point-to-point structure between two pixels rather than the high-order structure of multiple pixels, which is contradict with the real feature distribution of ground object. And the nonlinear property of HSI also brings challenge for precise structural representation in GCN. To tackle these problems, this work proposes a structure preserved hypergraph convolution network (SPHGCN). It first builds a multiple neighborhood reconstruction (MNR) model to reveal the essential resemblance of multiple pixels in nonlinear spectral feature space. With the high-order structure, SPHGCN designs the hypergraph convolution operation for irregular feature aggregation among similar pixels from different regions, which achieves more discriminative features from multiple pixel nodes. Meanwhile, a structure preservation layer is built to optimize the distribution of convolutional features under the guidance of high-order structure. Moreover, SPHGCN integrates local regular convolution and irregular hypergraph convolution to learn the structured semantic feature of HSI. This strategy breaks the boundary restriction in traditional convolution and aggregates semantic feature across different image patches. Experiments on three HSI data sets indicate that SPHGCN outperforms a few state-of-the-art methods for HSI classification.

Progressive Contextual Aggregation Empowered by Pixel-wise Confidence Scoring for Image Inpainting.

Image inpainting methods leverage the similarity of adjacent pixels to create alternative content. However, as the invisible region becomes larger, the pixels completed in the deeper hole are difficult to infer from the surrounding pixel signal, which is more prone to visual artifacts. To help fill this void, we adopt an alternative progressive hole-filling scheme that hierarchically fills the corrupted region in the feature and image spaces. This technique allows us to utilize reliable contextual information of the surrounding pixels, even for large hole samples, and then gradually complete the details as the resolution increases. For a more realistic representation of the completed region, we devise a pixel-wise dense detector. By distinguishing each pixel as whether it is a masked region or not, and passing the gradient to all resolutions, the generator further enhances the potential quality of the compositing. Furthermore, the completed images at different resolutions are then merged using a proposed structure transfer module (STM) that incorporates fine-grained local and coarse-grained global interactions. In this new mechanism, each completed image at the different resolutions attends its closest composition at fine granularity adjacent image and thus can capture the global continuity by interacting both short- and long-range dependencies. By comparing our solutions qualitatively and quantitatively with state-of-the-art methods, we conclude that our model exhibits a significantly improved visual quality, even in the case of large holes.

Adaptive hypergraph superpixels

Superpixel segmentation, which amounts to partitioning an image into a number of superpixels each of which is a set of pixels sharing common visual meanings, requires specific needs for different computer vision tasks. Graph based methods, as a kind of popular superpixel segmentation method, regard an image as a weighted graph whose nodes correspond to pixels of the image, and partition all pixels into superpixels according to the similarity between pixels over various feature spaces. Despite their improvement of the performance of segmentation, these methods ignore high-order relationship between them incurred from either locally neighboring pixels or structured layout of the image. Moreover, they measure the similarity of pairwise pixels using Gaussian kernel where a robust radius parameter is difficult to find for pixels which exhibit multiple features (e.g., texture, color, brightness). In this paper, we propose an adaptive hypergraph superpixel segmentation (AHS) of intensity images for solving both issues. AHS constructs a hypergraph by building the hyperedges with an adaptive neighborhood scheme, which explores an intrinsic relationship of pixels. Afterwards, AHS encodes the relationship between pairwise pixels using characteristics of current two pixels as well as their neighboring pixels defined by hyperedges. Essentially, AHS models the relationship of pairwise pixels in a high-order group fashion while graph based methods evaluate it in a one-vs-one fashion. Experiments on four datasets demonstrate that AHS achieves higher or comparable performance compared with state-of-the-art methods.

Geometry-Injected Image-Based Point Cloud Semantic Segmentation

Image-based methods have replicated the success from 2D domain to 3D point cloud semantic segmentation. However, when we directly apply 2D techniques to the projected pseudo-image, inherent differences between the point cloud and the image cause geometric distortion. This paper analyzes the geometric distortion between the point cloud and the pseudo-image, including truncation, dislocation, and hole. To ensure geometric fidelity, we propose the Geometry-injected Image-based point cloud semantic segmentation Network (GINet). We design a Cyclic Convolution to optimize the convolution operation, dealing with truncation. For dislocation and hole, we propose Dual Geometric Constraints, including Local Spatial Attention and Local Affinity Regularization, to incorporate the geometric information into semantic feature learning. Local Spatial Attention generates an attention map from the point coordinates to modulate the feature map before convolution. Local Affinity Regularization supervises the semantic similarity of pixels in the convolution kernel range. GINet rectifies the geometric distortion with these mechanisms while taking advantage of the successful 2D semantic segmentation methods. Quantitative and qualitative experiments on SemanticKITTI and SemanticPOSS demonstrate the effectiveness of GINet.

Macroscopic-and-Microscopic Rain Streaks Disentanglement Network for Single-Image Deraining.

Single-image deraining aims to restore the image that is degraded by the rain streaks, where the long-standing bottleneck lies in how to disentangle the rain streaks from the given rainy image. Despite the progress made by substantial existing works, several crucial questions - e.g., How to distinguish rain streaks and clean image, while how to disentangle rain streaks from low-frequency pixels, and further prevent the blurry edges - have not been well investigated. In this paper, we attempt to solve all of them under one roof. Our observation is that rain streaks are bright stripes with higher pixel values that are evenly distributed in each color channel of the rainy image, while the disentanglement of the high-frequency rain streaks is equivalent to decreasing the standard deviation of the pixel distribution for the rainy image. To this end, we propose a self-supervised rain streaks learning network to characterize the similar pixel distribution of the rain streaks from a macroscopic viewpoint over various low-frequency pixels of gray-scale rainy images, coupling with a supervised rain streaks learning network to explore the specific pixel distribution of the rain streaks from a microscopic viewpoint between each paired rainy and clean images. Building on this, a self-attentive adversarial restoration network comes up to prevent the further blurry edges. These networks compose an end-to-end Macroscopic-and-Microscopic Rain Streaks Disentanglement Network, named [Formula: see text]RSD-Net, to learn rain streaks, which is further removed for single image deraining. The experimental results validate its advantages on deraining benchmarks against the state-of-the-arts. The code is available at: https://github.com/xinjiangaohfut/MMRSD-Net.

Restoration of Dental Radiographs Corrupted by Quantum Noise via Noise-adaptive Sub-global Weighted Sum of Radiometrically Similar Pixels

Restoration of Dental Radiographs Corrupted by Quantum Noise via Noise-adaptive Sub-global Weighted Sum of Radiometrically Similar Pixels

Multiscale and Multisubgraph-Based Segmentation Method for Ocean Remote Sensing Images

Interpreting ocean remote sensing images is still a challenge is worth studying because they can carry valuable information for various important applications. Due to the absence of labeled datasets, unsupervised object-based image analysis (OBIA) methods provide an effective solution to understand remote sensing images with its advantage of grouping local similar pixels into a homogeneous area. However, ocean remote sensing images usually have the characteristics of large size, large background, and coexisting of large and small objects, which result in previous OBIA methods easily falling into the difficulty of accurately segmenting the large and small objects at the same time, and the dilemma of time-consuming computation. To solve this problem, a novel multi-scale and multi-subgraph (MSMSG)-based image segmentation method is presented in this paper. First, a coarse-to-fine superpixel generation method is designed to generate optimal superpixels, which can not only solve the problem of coexisting large objects and small objects but also the problem of manually setting the initial segmentation number. Second, the proposed background removal strategy helps to eliminate the trouble of large background areas in ocean remote sensing images. Third, a multi-subgraph is constructed with the help of background removal. Finally, the multi-scale and multi-subgraph merging strategy is addressed to group all similar superpixels into a same cluster, which not only reduces the useless computation of nonadjacent superpixels but also avoids segmentation errors with the same scale. Experiments conducted on three different datasets show that the proposed segmentation method is high-performance and high-efficiency.

Front-end Electronics for Timing with pico-second precision using 3D Trench Silicon Sensors

The next generation of collider experiments require tracking detectors with extreme performance capabilities in terms of spatial resolution (tens of µm), radiation hardness (1017 1 MeV neq/cm2) and timing resolution (tens of ps). 3D silicon sensors, recently developed within the TimeSPOT initiative, offer a viable solution to cope with such demanding requirements. In order to accurately characterize the timing performance of these new sensors, several read-out boards, based on discrete active components, have been designed, assembled, and tested. The same electronics is also suitable for characterization of similar pixel sensors whenever timing performance in the order and below 10 ps is a requirement. This paper describes the general characteristics needed by front-end electronics to exploit solid-state sensors with fast timing capabilities and in particular, showcases the performance of the developed electronics in the testing and characterization of fast 3D silicon sensors.

Light Field Image Super-Resolution Network via Joint Spatial-Angular and Epipolar Information

This paper proposes a novel efficient convolutional neural network (CNN) for light field (LF) spatial super-resolution (SR). Due to the high dimensionality of LF data, it is important to decompose the original 4D LF data structure into a lower data subspace helping to reduce memory and computational costs. In this context, we propose a novel hybrid CNN structure working on different 2D data representations to fully handle the 4D structure. Our proposed network can simultaneously exploit information from three different types of 4D LF representations: spatial, angular, and epipolar plane image (EPI) information. Based on pixel similarities of each data subspace, we propose an inter-intra spatial feature extractor (II-SFE) and an inter-intra angular feature extractor (II-AFE) that can exploit the pixel correlations among the spatial-angular domain. Moreover, based on the correlated epipolar geometry among pixels, we also propose a multi-orientation epipolar feature extractor (MO-EFE) by decomposing an LF image into four orientation stacks of views and feeding them into different network branches to learn rich sub-pixel information in the EPI domain. The extracted features of each 2D subspace are integrated by our proposed attention fusion module (AFM), which adaptively rescales the embedding weights based on their correlations. Experimental results on both small- and wide-baseline benchmark LF datasets confirm the effectiveness of the proposed method compared to the well-known state-of-the-art methods in terms of objective and subjective performance.

Toward Semantic Communications: Deep Learning-Based Image Semantic Coding

Semantic communications has received growing interest since it can remarkably reduce the amount of data to be transmitted without missing critical information. Most existing works explore the semantic encoding and transmission for text and apply techniques in Natural Language Processing (NLP) to interpret the meaning of the text. In this paper, we conceive the semantic communications for image data that is much more richer in semantics and bandwidth sensitive. We propose an reinforcement learning based adaptive semantic coding (RL-ASC) approach that encodes images beyond pixel level. Firstly, we define the semantic concept of image data that includes the category, spatial arrangement, and visual feature as the representation unit, and propose a convolutional semantic encoder to extract semantic concepts. Secondly, we propose the image reconstruction criterion that evolves from the traditional pixel similarity to semantic similarity and perceptual performance. Thirdly, we design a novel RL-based semantic bit allocation model, whose reward is the increase in rate-semantic-perceptual performance after encoding a certain semantic concept with adaptive quantization level. Thus, the task-related information is preserved and reconstructed properly while less important data is discarded. Finally, we propose the Generative Adversarial Nets (GANs) based semantic decoder that fuses both locally and globally features via an attention module. Experimental results demonstrate that the proposed RL-ASC is noise robust and could reconstruct visually pleasant and semantic consistent image in low bit rate condition.

A Novel Multi-Scale Feature Fusion-Based 3SCNet for Building Crack Detection

Crack detection at an early stage is necessary to save people’s lives and to prevent the collapse of building/bridge structures. Manual crack detection is time-consuming, especially when a building structure is too high. Image processing, machine learning, and deep learning-based methods can be used in such scenarios to build an automatic crack detection system. This study uses a novel deep convolutional neural network, 3SCNet (3ScaleNetwork), for crack detection. The SLIC (Simple Linear Iterative Clustering) segmentation method forms the cluster of similar pixels and the LBP (Local Binary Pattern) finds the texture pattern in the crack image. The SLIC, LBP, and grey images are fed to 3SCNet to form pool of feature vector. This multi-scale feature fusion (3SCNet+LBP+SLIC) method achieved the highest sensitivity, specificity, an accuracy of 99.47%, 99.75%, and 99.69%, respectively, on a public historical building crack dataset. It shows that using SLIC super pixel segmentation and LBP can improve the performance of the CNN (Convolution Neural Network). The achieved performance of the model can be used to develop a real-time crack detection system.

Target anisotropic scattering deduction model using multi-aspect SAR data

Typical targets in SAR observing scenes are characterized by anisotropy, which represents backscattering information changing with the observing angle. Anisotropic scattering can be used to study the structural characteristics of targets and the response differences between structures. In actual situations, due to flight conditions and other reasons, only partial observation information can be obtained, so it is important to use limited data to deduce remaining backscattering information for target analysis. In our article, the scattering deduction model based on matrix completion is established, including scattering characteristic analysis. The multi-aspect data is transferred from range and azimuth in conventional SAR image to pixel dimension and angle dimension for analysis. At the same time, the pixel similarity matrix is used as a feature in pixel dimension, and one-dimension back-amplitude scattering curve with angle is established as a feature matrix in angle dimension. In the process of modeling existing data, two corresponding decomposition matrices are generated with the help of collaborative filtering idea. Finally, a four-matrix combination for anisotropy deduction is formed. Ku-band multi-aspect data from autonomous experimental flights is used for algorithm. In the experiment, the scattering deduction of the remaining aspect sequence was carried out based on obtaining partial aspect sequence images, and the strong scattering regions of the target point and sub-aperture details were compared. The edge information and structure map of the target were reconstructed using scattering deduction results.

Detecting Lung Cancer Region from CT Image Using Meta-Heuristic Optimized Segmentation Approach

Lung tumor detection using computer-aided modeling improves the accuracy of detection and clinical recommendation precision. An optimal tumor detection requires noise reduced computed tomography (CT) images for pixel classification. In this paper, the butterfly optimization algorithm-based [Formula: see text]-means clustering (BOAKMC) method is introduced for reducing CT image segmentation uncertainty. The introduced method detects the overlapping features for optimal edge classification. The best-fit features are first trained and verified for their similarity. The clustering process recurrently groups the feature matched pixels into clusters and updates the centroid based on further classifications. In this classification process, the uncertain pixels are identified and mitigated in the tumor detection analysis. The best-fit features are used to train local search instances in the BOA process, which influences the similar pixel grouping in the uncertainty detection process. The proposed BOAKMC improves accuracy and precision by 10.2% and 13.39% and reduces classification failure and time by 11.29% and 11.52%, respectively.