Pixel Features Research Articles

A Method to Reduce the Intra-Frame Prediction Complexity of HEVC Based on D-CNN

Among a series of video coding standards jointly developed by ITU-T, VCEG, and MPEG, high-efficiency video coding (HEVC) is one of the most widely used video coding standards today. Therefore, it is still necessary to further reduce the coding complexity of HEVC. In the HEVC standard, a flexible partitioning procedure entitled “quad-tree partition” is proposed to significantly improve the coding efficiency, which, however, leads to high coding complexity. To reduce the coding complexity of the intra-frame prediction, this paper proposes a scheme based on a densely connected convolution neural network (D-CNN) to predict the partition of coding units (CUs). Firstly, a densely connected block was designed to improve the efficiency of the CU partition by fully extracting the pixel features of CTU. Then, efficient channel attention (ECA) and adaptive convolution kernel size were applied to a fast CU partition for the first time to capture the information of the D-CNN convolution channels. Finally, a threshold optimization strategy was formulated to select the best threshold for each depth to further balance the computation complexity of video coding and the performance of RD. The experimental results show that the proposed method reduces the encoding time of HEVC by 60.14%, with a negligible reduction in RD performance, which is better than the existing fast partitioning methods.

LiteST-Net: A Hybrid Model of Lite Swin Transformer and Convolution for Building Extraction from Remote Sensing Image

Extracting building data from remote sensing images is an efficient way to obtain geographic information data, especially following the emergence of deep learning technology, which results in the automatic extraction of building data from remote sensing images becoming increasingly accurate. A CNN (convolution neural network) is a successful structure after a fully connected network. It has the characteristics of saving computation and translation invariance with improved local features, but it has difficulty obtaining global features. Transformers can compensate for the shortcomings of CNNs and more effectively obtain global features. However, the calculation number of transformers is excessive. To solve this problem, a Lite Swin transformer is proposed. The three matrices Q, K, and V of the transformer are simplified to only a V matrix, and the v of the pixel is then replaced by the v with the largest projection value on the pixel feature vector. In order to better integrate global features and local features, we propose the LiteST-Net model, in which the features extracted by the Lite Swin transformer and the CNN are added together and then sampled up step by step to fully utilize the global feature acquisition ability of the transformer and the local feature acquisition ability of the CNN. The comparison experiments on two open datasets are carried out using our proposed LiteST-Net and some classical image segmentation models. The results show that compared with other networks, all metrics of LiteST-Net are the best, and the predicted image is closer to the label.

Feature Representation for High‐resolution Clothed Human Reconstruction

AbstractDetailed and accurate feature representation is essential for high‐resolution reconstruction of clothed human. Herein we introduce a unified feature representation for clothed human reconstruction, which can adapt to changeable posture and various clothing details. The whole method can be divided into two parts: the human shape feature representation and the details feature representation. Specifically, we firstly combine the voxel feature learned from semantic voxel with the pixel feature from input image as an implicit representation for human shape. Then, the details feature mixed with the clothed layer feature and the normal feature is used to guide the multi‐layer perceptron to capture geometric surface details. The key difference from existing methods is that we use the clothing semantics to infer clothed layer information, and further restore the layer details with geometric height. We qualitative and quantitative experience results demonstrate that proposed method outperforms existing methods in terms of handling limb swing and clothing details. Our method provides a new solution for clothed human reconstruction with high‐resolution details (style, wrinkles and clothed layers), and has good potential in three‐dimensional virtual try‐on and digital characters.

Encryption of Dynamic Areas of Images in Video based on Certain Geometric and Color Shapes

The paper is devoted to the search for new approaches to encrypting selected objects in an image. Videos were analyzed, which were divided into frames, and in each video frame, the necessary objects were detected for further encryption. Images of objects with a designated geometric shape and color characteristics of pixels were considered. To select objects, a method was used based on the calculation of average values, the analysis of which made it possible to determine the convergence with the established image. Dividing the selected field into subregions with different shapes solves the problem of finding objects of the same type with different scales. In addition, the paper considers the detection of moving objects. The detection of moving objects is carried out based on determining the frame difference in pixel codes in the form of a rectangular shape. Cellular automata technology was used for encryption. The best results were shown by the transition rules of elementary cellular automata, such as: 90, 105, 150, and XOR function. The use of cellular automata technologies made it possible to use one key sequence to encrypt objects on all video frames of the video. Encryption results are different for the same objects located in different places of the same video frame and different video frames of the video sequence. The video frame image is divided into bit layers, the number of which is determined by the length of the code of each pixel. Each bit layer is encrypted with the same evolution, which is formed by one initial key bit sequence. For each video frame, a different part of the evolution is used, as well as for each detected object in the image. This approach gives different results for any objects that have a different location both on the video frame image and in different video frames. The described methods allow you to automate the process of detecting objects on video and encrypting them.

Macroalgae monitoring from satellite optical images using Context-sensitive level set (CSLS) model

Automatically detecting macroalgae from optical remote sensing images has been a long-standing problem in ocean monitoring. Existing operational spectral-based techniques tend to assign a threshold to an indicator such as Normalized Difference Vegetation Index (NDVI) to determine whether or not macroalgae exists in a pixel. This thresholding scheme relies on individual pixel features and cannot address the environment variability among different images or even across one single image. To address these limitations, we develop the unsupervised classification method taking the ecological indicator of NDVI as input by exploiting a region-based level set driven by context-sensitive energy function for macroalgae detection from optical remote sensing images. Quantitative and qualitative evaluations based on satellite optical images with various imaging time and spatial resolutions (16–250 m) from GF-1, GF-4, Landsat-8, and Aqua demonstrate the effectiveness and robustness for Ulva prolifera detection. The model is proved to be superior to the traditional NDVI thresholding method and state-of-the-art level set methods in terms of higher accuracy (Kappa coefficients 0.87–0.92) and automation (without artificial threshold/parameter adjustment). Moreover, the model has the advantage of accommodating complex environmental conditions (such as different water turbidity, thin clouds and sun glint), and can be utilized for detection of Sargassum bloom. The model could facilitate the accurate extraction of macroalgae information and thus serve disaster prevention and control. It may also be used to assist high-accuracy training data preparation for supervised classification and detect the objects with obvious texture and spectral characteristics (such as oil spills) based on customized bands combination.

Detection of Moving Object Using Superpixel Fusion Network

Moving object detection is still a challenging task in complex scenes. The existing methods based on deep learning mainly use U-Nets and have achieved amazing results. However, they ignore the local continuity between pixels. In order to solve this problem, a method based on a superpixel fusion network (SF-Net) is proposed in this article. First, the median filter is used to extract the candidate foreground (called pixel features ) and the image sequence is segmented by superpixel. Then, the histogram features (called superpixel features ) of the candidate foreground superpixels are extracted. Next, the pixel features and the superpixel features are the inputs of SF-Net, respectively. Experiments show the effectiveness of SF-Net on 34 image sequences and the average F-measure reaches 0.84. SF-Net can remove more background noise and has stronger expression ability than a network with the same depth.

An improved method for retrieving aerosol optical depth over Ebinur Lake Basin from Gaofen-1

Lake Ebinur is the largest saltwater lake in Xinjiang. In recent years, the exposed dry lake bottom surface has become a remarkable source of salt and dust owing to the desiccation of the lake. Given the importance of the lake basin to human livelihoods, comprehensive monitoring of basin behavior is required. Remote sensing offers numerous novel approaches to monitoring the spatial and temporal distribution of aerosols. During this study, high spatial resolution aerosol optical depth (AOD) inversion at Lake Ebinur was still in the gap stage. The wide-field-of-view (WFV) data from four cameras on the GaoFen-1 (GF-1) satellite, launched by China in April 2013, offers high spatial and temporal resolution and excellent potential for AOD estimation. Owing to the shortwave infrared band (SWIR) deficiency and the complexity of surface reflectance, the coefficient of variation (CV) was proposed as a measure of surface reflectance to develop a surface reflectance database for land cover characteristics in the Ebinur Lake Basin. The significant improvements of the algorithm include (1) the design of an accurate and universal cloud detection algorithm through the blue and red bands. (2) The CV was used to assess the robustness of surface reflectance under long time series. After dividing the surface reflectance image pixels with low robustness, a reasonable value was assigned according to the surface reflectance variation characteristics of the image pixels. (3) Aerosol models were determined from the long-term measurements of the CIMEL solar photometer (CE318), and the validation of ground-based observations showed good correlation that WFV/GF-1 AOD was comparable to CE318 AOD (R = 0.705; root mean square error (RMSE) = 0.098; mean absolute error (MAE) = 0.078), and the smaller MAE and RMSE confirmed the feasibility of inversion of AOD by this method.

Language-Augmented Pixel Embedding for Generalized Zero-Shot Learning

Zero-shot Learning (ZSL) aims to recognize novel classes through seen knowledge. The canonical approach to ZSL leverages a visual-to-semantic embedding to map the global features of an image sample to its semantic representation. These global features usually overlook the fine-grained information which is vital for knowledge transfer between seen and unseen classes, rendering these features sub-optimal for ZSL task, especially the more realistic Generalized Zero-shot Learning (GZSL) task where global features of similar classes could hardly be separated. To provide a remedy to this problem, we propose Language-Augmented Pixel Embedding (LAPE) that directly bridges the visual and semantic spaces in a pixel-based manner. To this end, we map the local features of each pixel to different attributes and then extract each semantic attribute from the corresponding pixel. However, the lack of pixel-level annotation conduces to an inefficient pixel-based knowledge transfer. To mitigate this dilemma, we adopt the text information of each attribute to augment the local features of image pixels which are related to the semantic attributes. Experiments on four ZSL benchmarks demonstrate that LAPE outperforms current state-of-the-art methods. Comprehensive ablation studies and analyses are provided to dissect what factors lead to this success.

Research and Application of Generative-Adversarial-Network Attacks Defense Method Based on Federated Learning

In recent years, Federated Learning has attracted much attention because it solves the problem of data silos in machine learning to a certain extent. However, many studies have shown that attacks based on Generative Adversarial Networks pose a great threat to Federated Learning. This paper proposes Defense-GAN, a defense method against Generative Adversarial Network attacks under Federated Learning. Under this method, the attacker cannot learn the real image data distribution. Each Federated Learning participant uses SHAP to explain the model and masks the pixel features that have a greater impact on classification and recognition in their respective image data. The experimental results show that while attacking the federated training model using masked images, the attacker cannot always obtain the ground truth of the images. At the same time, this paper also uses CutMix to improve the generalization ability of the model, and the obtained model accuracy is only 1% different from that of the model trained with the original data. The results show that the defense method proposed in this paper can not only resist Generative Adversarial Network attacks in Federated Learning and protect client privacy, but also ensure that the model accuracy of the Federated model will not be greatly affected.

A differential correction based shadow removal method for real-time monitoring.

Shadow removal is an important issue in the field of motion object surveillance and automatic control. Although many works are concentrated on this issue, the diverse and similar motion patterns between shadows and objects still severely affect the removal performance. Constrained by the computational efficiency in real-time monitoring, the pixel feature based methods are still the main shadow removal methods in practice. Following this idea, this paper proposes a novel and simple shadow removal method based on a differential correction calculation between the pixel values of Red, Green and Blue channels. Specifically, considering the fact that shadows are formed because of the occlusion of light by objects, all the reflected light will be attenuated. Hence there will be a similar weakening trends in all Red, Green and Blue channels of the shadow areas, but not in the object areas. These trends can be caught by differential correction calculation and distinguish the shadow areas from object areas. Based on this feature, our shadow removal method is designed. Experiment results verify that, compared with other state-of-the-art shadow removal methods, our method improves the average of object and shadow detection accuracies by at least 10% in most of the cases.

Weakly supervised semantic segmentation of histological tissue via attention accumulation and pixel-level contrast learning

Objective. Histopathology image segmentation can assist medical professionals in identifying and diagnosing diseased tissue more efficiently. Although fully supervised segmentation models have excellent performance, the annotation cost is extremely expensive. Weakly supervised models are widely used in medical image segmentation due to their low annotation cost. Nevertheless, these weakly supervised models have difficulty in accurately locating the boundaries between different classes of regions in pathological images, resulting in a high rate of false alarms Our objective is to design a weakly supervised segmentation model to resolve the above problems. Approach. The segmentation model is divided into two main stages, the generation of pseudo labels based on class residual attention accumulation network (CRAANet) and the semantic segmentation based on pixel feature space construction network (PFSCNet). CRAANet provides attention scores for each class through the class residual attention module, while the Attention Accumulation (AA) module overlays the attention feature maps generated in each training epoch. PFSCNet employs a network model containing an inflated convolutional residual neural network and a multi-scale feature-aware module as the segmentation backbone, and proposes dense energy loss and pixel clustering modules are based on contrast learning to solve the pseudo-labeling-inaccuracy problem. Main results. We validate our method using the lung adenocarcinoma (LUAD-HistoSeg) dataset and the breast cancer (BCSS) dataset. The results of the experiments show that our proposed method outperforms other state-of-the-art methods on both datasets in several metrics. This suggests that it is capable of performing well in a wide variety of histopathological image segmentation tasks. Significance. We propose a weakly supervised semantic segmentation network that achieves approximate fully supervised segmentation performance even in the case of incomplete labels. The proposed AA and pixel-level contrast learning also make the edges more accurate and can well assist pathologists in their research.

Robust spectral clustering integrated nonlocal information blocks for hyperspectral image classification

Hyperspectral image (HSI) clustering is a challenging task due to its containing rich spectral information and spatial features. Subspace clustering has been used to explore the intrinsic connections between data points successfully. Graph convolutional subspace clustering (GCSC) was used to achieve robust HSI clustering. The model remaps the self-expressions of the data to non-Euclidean domains to generate a robust graph embedding dictionary. The efficient kernel graph convolutional subspace clustering model utilizes a subspace clustering model with the Frobenius norm and a Gaussian kernel function to achieve a globally optimal closed-form solution that is easy to implement, train, and apply. To reduce the impact of image noise on the segmentation effect, we consider spatial nonlocal self-similarity using nonlocal information blocks based on the fact that the spectral features of HSIs lie in low-dimensional subspaces. At the same time, the three-dimensional tensor generated by the nonlocal similar image blocks is estimated using a robust statistic function. Principal component analysis is used to remove the redundant information of hyperspectral data and extract the main features of pixels. Then a robust statistical function and nonlocal information block are used to construct the affinity matrix. The redundant information prevalent in the whole natural image is used for denoising, that is, the image block is taken as the unit to find similar areas in the image, then these regions are weighted and averaged. Since all pixels in the image are used, the noise can be better filtered. Finally, the constructed affinity matrix is applied to spectral clustering to obtain better clustering results. The model used is called nonlocal mean robust statistical kernel graph convolutional subspace clustering (N-RKGCSC). Experimental results on Salinas, Indian Pines, Pavia Center, and Pavia University datasets showed that the N-RKGCSC model could eliminate the interference of additive and multiplicative noise well, achieving better segmentation performance and better noise immunity.

Reduced volume of diabetic pancreatic islets in rodents detected by synchrotron X-ray phase-contrast microtomography and deep learning network

The pancreatic islet is a highly structured micro-organ that produces insulin in response to rising blood glucose. Here we develop a label-free and automatic imaging approach to visualize the islets in situ in diabetic rodents by the synchrotron radiation X-ray phase-contrast microtomography (SRμCT) at the ID17 station of the European Synchrotron Radiation Facility. The large-size images (3.2mm×15.97mm) were acquired in the pancreas in STZ-treated mice and diabetic GK rats. Each pancreas was dissected by 3000 reconstructed images. The image datasets were further analysed by a self-developed deep learning method, AA-Net. All islets in the pancreas were segmented and visualized by the three-dimension (3D) reconstruction. After quantifying the volumes of the islets, we found that the number of larger islets (=>1500 μm3) was reduced by 2-fold (wt 1004±94 vs GK 419±122, P<0.001) in chronically developed diabetic GK rat, while in STZ-treated diabetic mouse the large islets were decreased by half (189±33 vs 90±29, P<0.001) compared to the untreated mice. Our study provides a label-free tool for detecting and quantifying pancreatic islets in situ. It implies the possibility of monitoring the state of pancreatic islets in vivo diabetes without labelling.

TransAttention U-Net for Semantic Segmentation of Poppy

This work represents a new attempt to use drone aerial photography to detect illegal cultivation of opium poppy. The key of this task is the precise segmentation of the poppy plant from the captured image. To achieve segmentation mask close to real data, it is necessary to extract target areas according to different morphological characteristics of poppy plant and reduce complex environmental interference. Based on RGB images, poppy plants, weeds, and background regions are separated individually. Firstly, the pixel features of poppy plant are enhanced using a hybrid strategy approach to augment the too-small samples. Secondly, the U-Shape network incorporating the self-attention mechanism is improved to segment the enhanced dataset. In this process, the multi-head self-attention module is enhanced by using relative position encoding to deal with the special morphological characteristics between poppy stem and fruit. The results indicated that the proposed method can segmented out the poppy plant precisely.

A gamma radiation scene image clarification method based on spatiotemporal combination

For the problems of complex noise and color cast in the images captured by complementary metal oxide semiconductor image sensors in the gamma radiation environment, this paper proposed a joint temporal correlation denoising and color cast correction approach for the gamma radiation scene image clarification. Concretely, we first adaptively stretch the difference between noise and background to improve the salience of weak noise, making noise detection results more accurate. Secondly, the temporal transient characteristics of gamma radiation noise are utilized for noise detection, and these noisy pixels are repaired by exploiting the temporal correlation characteristics of background pixels. Lastly, the Bayer-Guided auto white balance method is proposed to solve the problem of dysregulated color in the gamma radiation scene image, and the color level remapping approach is applied to eliminate the problem of insufficient dynamic range. Extensive experiments are carried out on the images captured from the Co60 gamma radiation scene. Compared with the source images, our method improves the peak signal to noise ratio by 6.3(dB) and structural similarity by 0.34. Experimental results demonstrate that the proposed method enjoys state-of-the-art performance in improving the clarity of gamma radiation scene images.

Super-resolution fusion optimization for poultry detection: a multi-object chicken detection method.

Accurate poultry detection is crucial for studying poultry behavior using computer vision and video surveillance. However, in free-range farming environments, detecting chickens can often be challenging due to their small size and mutual occlusion. The current detection algorithms exhibit a low level of accuracy, with a high probability of false and missed detections. To address this, we proposed a multi-object chicken detection method named Super-resolution Chicken Detection, which utilizes super-resolution fusion optimization. The algorithm employs the residual-residual dense block to extract image features and used a generative adversarial network to compensate for the loss of details during deep convolution, producing high-resolution images for detection. The proposed algorithm was validated with the B1 data set and the MC1 multi-object data set, demonstrating that the reconstructed images possessed richer pixel features compared to original images, specifically it improved detection accuracy and reduced the number of missed detections. The structural similarity of the reconstructed images was 99.9%, and the peak signal-to-noise ratio was above 30. The algorithm improved the Average Precision50:95 of all You Only Look Once Version X (YOLOX) models, with the largest improvement for the B1 data set with YOLOX-Large (+6.3%) and for the MC1 data set with YOLOX-Small (+4.1%). This was the first time a super-resolution reconstruction technique was applied to multi-object poultry detection. Our method will provide a fresh approach for future poultry researchers to improve the accuracy of object detection using computer vision and video surveillance.

Dynamic Super-Pixel Normalization for Robust Hyperspectral Image Classification

Deep neural networks (DNNs) have underpinned most of recent progress of hyperspectral image (HSI) classification. One premise of their success lies on the high image quality without noise corruption. However, due to the limitation of imaging sensor and imaging conditions, HSIs captured in practice inevitably suffer from random noise, which will degrade the generalization performance and robustness of most existing DNN based methods. In this study, we propose a dynamic super-pixel normalization based DNN for HSI classification, which can adaptively relieve the negative effect caused by various types of noise corruption and improve the generalization performance. To achieve this goal, we propose a dynamic super-pixel normalization module, for a given super-pixel which normalizes the inner pixel features using parameters dynamically generated based on themselves. By doing this, such a module enables adaptively restoring the similarity among pixels within the super-pixel corrupted by random noise through aligning their feature distribution, thus enhancing the generalization performance on noisy HSI. Moreover, it can be directly plugged into any other existing DNN architectures. To appropriately train the proposed DNN model, we further present a semi-supervised learning framework, which integrates the cross entropy loss and Kullback-Leibler divergence loss on labeled samples with the infomation entropy loss on the unlabeled samples for joint learning to well sidestep over-fitting. Experiments on three benchmark HSI classification datasets demonstrate the advantages of the proposed method over several state-of-the-art competitors in handling HSIs under different types of noise corruption.

Hyperspectral Image Classification Using a Superpixel–Pixel–Subpixel Multilevel Network

Hyperspectral images (HSIs) often contain irregular ground cover with mixed spectral features and noise, which makes it challenging to identify the ground cover using only pixel features, superpixel features, or a combination of both. To alleviate the above problem, this paper proposes a superpixel-pixel-subpixel multilevel network (SPSM), which compensates for the insufficiencies of the different levels and decrease the information loss. For arbitrary irregular regions, superpixel features are simulated as network nodes using a graph convolutional network (GCN) to capture the spatial texture structure of the HSI, which improves the smooth classification results of local regions while facilitating the identification of different vegetation features in the region. Additionally, the global attention module (GAM) learns local regular regions based on pixel-level features to extend the global interactive representation capability and reduce information loss. To overcome spectral mixing and enhance material discrimination, the normalized attention module (NAM) is used to suppress unimportant subpixel information and identify and remove irrelevant details, thereby improving the identification of critical features that differentiate different materials. Finally, the three features are fused to build a SPSM classification framework to improve robustness to overfitting, reduce computational complexity, and facilitate target recognition. Experimental results on four HSI datasets demonstrate that the method is more capable of recognizing detailed features than other advanced comparison methods.

Video anomaly detection by fusing self-attention and autoencoder

Video anomaly detection by fusing self-attention and autoencoder

CPP-Net: Context-aware Polygon Proposal Network for Nucleus Segmentation.

Nucleus segmentation is a challenging task due to the crowded distribution and blurry boundaries of nuclei. To differentiate between touching and overlapping nuclei, recent approaches have represented nuclei in the form of polygons, and have accordingly achieved promising performance. Each polygon is represented by a set of centroid-to-boundary distances, which are in turn predicted by features of the centroid pixel for a single nucleus. However, the use of the centroid pixel alone does not provide sufficient contextual information for robust prediction and therefore affects the segmentation accuracy. To address this problem, we propose a Context-aware Polygon Proposal Network (CPP-Net) for nucleus segmentation. First, we sample a point set rather than a single pixel within each cell for distance prediction; this strategy substantially enhances the contextual information and thereby improves the prediction robustness. Second, we propose a Confidence-basedWeighting Module, which adaptively fuses the predictions from the sampled point set. Third, we introduce a novel Shape-Aware Perceptual (SAP) loss that constrains the shape of the predicted polygons. This SAP loss is based on an additional network that is pre-trained by means of mapping the centroid probability map and the pixel-to-boundary distance maps to a different nucleus representation. Extensive experiments demonstrate the effectiveness of each component in the proposed CPP-Net. Finally, CPP-Net is found to achieve state-of-the-art performance on three publicly available databases, namely DSB2018, BBBC06, and PanNuke. The code of this paper will be released.