Pixel-wise Prediction Research Articles

Prototype-Based Semantic Segmentation.

Deep learning based semantic segmentation solutions have yielded compelling results over the preceding decade. They encompass diverse network architectures (FCN based or attention based), along with various mask decoding schemes (parametric softmax based or pixel-query based). Despite the divergence, they can be grouped within a unified framework by interpreting the softmax weights or query vectors as learnable class prototypes. In light of this prototype view, we reveal inherent limitations within the parametric segmentation regime, and accordingly develop a nonparametric alternative based on non-learnable prototypes. In contrast to previous approaches that entail the learning of a single weight/query vector per class in a fully parametric manner, our approach represents each class as a set of non-learnable prototypes, relying solely upon the mean features of training pixels within that class. The pixel-wise prediction is thus achieved by nonparametric nearest prototype retrieving. This allows our model to directly shape the pixel embedding space by optimizing the arrangement between embedded pixels and anchored prototypes. It is able to accommodate an arbitrary number of classes with a constant number of learnable parameters. Through empirical evaluation with FCN based and Transformer based segmentation models (i.e., HRNet, Swin, SegFormer, Mask2Former) and backbones (i.e., ResNet, HRNet, Swin, MiT), our nonparametric framework shows superior performance on standard segmentation datasets (i.e., ADE20 K, Cityscapes, COCO-Stuff), as well as in large-vocabulary semantic segmentation scenarios. We expect that this study will provoke a rethink of the current de facto semantic segmentation model design.

Exploring Generalizable Distillation for Efficient Medical Image Segmentation.

Efficient medical image segmentation aims to provide accurate pixel-wise predictions with a lightweight implementation framework. However, existing lightweight networks generally overlook the generalizability of the cross-domain medical segmentation tasks. In this paper, we propose Generalizable Knowledge Distillation (GKD), a novel framework for enhancing the performance of lightweight networks on cross-domain medical segmentation by generalizable knowledge distillation from powerful teacher networks. Considering the domain gaps between different medical datasets, we propose the Model-Specific Alignment Networks (MSAN) to obtain the domain-invariant representations. Meanwhile, a customized Alignment Consistency Training (ACT) strategy is designed to promote the MSAN training. Based on the domain-invariant vectors in MSAN, we propose two generalizable distillation schemes, Dual Contrastive Graph Distillation (DCGD) and Domain-Invariant Cross Distillation (DICD). In DCGD, two implicit contrastive graphs are designed to model the intra-coupling and inter-coupling semantic correlations. Then, in DICD, the domain-invariant semantic vectors are reconstructed from two networks (i.e., teacher and student) with a crossover manner to achieve simultaneous generalization of lightweight networks, hierarchically. Moreover, a metric named Fréchet Semantic Distance (FSD) is tailored to verify the effectiveness of the regularized domain-invariant features. Extensive experiments conducted on the Liver, Retinal Vessel and Colonoscopy segmentation datasets demonstrate the superiority of our method, in terms of performance and generalization ability on lightweight networks.

Information-Theoretic Exploration for Adaptive Robotic Grasping in Clutter Based on Real-Time Pixel-Level Grasp Detection

Robust robotic grasping in clutter still remains a challenging problem despite its great practical value. This paper presents an information-theoretic exploration approach that aims to improve the grasp estimation quality in complex environment. First, a light-weight grasp detector that composed by inverted residual blocks and a pyramid pooling module is proposed to make more accurate pixel-wise predictions in real-time, which are then projected to the workspace. To measure the uncertainty of estimations in the workspace, a two-dimensional grasp entropy (2D-GE) is defined and Gaussian process is applied to regress the variation of the information gain that approximated by 2D-GEs. Finally, guided by the information gain, depth cost, and distance cost, our approach is able to actively collect and fuse estimations from multiple informative viewpoints through the exploration that finally converges to a refined best grasp, resulting an improved grasping performance with a higher success rate and environmental adaptability in clutter. Simulation and robotic experiment are both performed to demonstrate the effectiveness of our method and compare it to baselines.

FCSN: Global Context Aware Segmentation by Learning the Fourier Coefficients of Objects in Medical Images.

The encoder-decoder model is a commonly used Deep learning (DL) model for medical image segmentation. Encoder-decoder models make pixel-wise predictions that focus heavily on local patterns. As a result, the predicted mask often fails to preserve the object's shape and topology, which requires an understanding of the image's global context. In this work, we propose a Fourier Coefficient Segmentation Network (FCSN)---a novel global context-aware DL model that segments an object by learning the complex Fourier Coefficients of the object's masks. The Fourier coefficients are calculated by integrating over the mask's contour. Hence, FCSN is naturally motivated to incorporate a broader image context when estimating the coefficients. The global context awareness of FCSN helps produce more accurate segmentation and is more robust to local perturbations, such as additive noise or motion blur. We compare FCSN on other state-of-the-art global context-aware models (UNet++, DeepLabV3+, UNETR) on 5 medical image segmentation tasks (ISIC_2018, RIM_CUP, RIM_DISC, PROSTATE, FETAL). When compared with UNETR, FCSN attains significantly lower Hausdorff scores with 19.14 (6%), 17.42 (6%), 9.16 (14%), 11.18 (22%), and 5.98 (6%) for ISIC_2018, RIM_CUP, RIM_DISC, PROSTATE, and FETAL tasks respectively. Moreover, FCSN is lightweight by discarding the decoder module. FCSN only requires 29.7 M parameters which are 75.6 M and 9.9 M fewer than UNETR and DeepLabV3+, respectively. FCSN attains inference/training speeds of 1.6 ms/img and 6.3 ms/img, which is 8 and 3 faster than UNet and UNETR. Our work is available at https://github.com/nus-morninlab/FCSN.

Fine-Grained Visual-Text Prompt-Driven Self-Training for Open-Vocabulary Object Detection.

Inspired by the success of vision-language methods (VLMs) in zero-shot classification, recent works attempt to extend this line of work into object detection by leveraging the localization ability of pretrained VLMs and generating pseudolabels for unseen classes in a self-training manner. However, since the current VLMs are usually pretrained with aligning sentence embedding with global image embedding, the direct use of them lacks fine-grained alignment for object instances, which is the core of detection. In this article, we propose a simple but effective fine-grained visual-text prompt-driven self-training paradigm for open-vocabulary detection (VTP-OVD) that introduces a fine-grained visual-text prompt adapting stage to enhance the current self-training paradigm with a more powerful fine-grained alignment. During the adapting stage, we enable VLM to obtain fine-grained alignment using learnable text prompts to resolve an auxiliary dense pixelwise prediction task. Furthermore, we propose a visual prompt module to provide the prior task information (i.e., the categories need to be predicted) for the vision branch to better adapt the pretrained VLM to the downstream tasks. Experiments show that our method achieves the state-of-the-art performance for open-vocabulary object detection, e.g., 31.5% mAP on unseen classes of COCO.

Processing of electrical resistivity tomography data using convolutional neural network in ERT-NET architectures

Electrical resistivity tomography (ERT) inversion has emerged as an effective method for predicting resistivity in complex geological structures. In most cases, traditional ERT inversion problems are posed as nonlinear optimization problems. Solving distribution resistivity inversion can be computationally challenging for two reasons: one is the significant cost of software and the other is the issue of local minima. The ERT-NET architecture was developed in this study to learn the parameter regression relationship between geophysical ERT datasets and subsurface models. We developed a novel convolutional neural network (CNN) technique that comprised of a fully connected network (ERT-INET) and a fully convolutional network (ERT-UNET); both train ambiguity information of the inverted resistivity based on processing ERT datasets. We also output our network segmentation of pixel-wise prediction for ERT-INET and structured prediction segmentation. The noise assessments of our network inversion were managed by employing depth of investigation (DOI) and statistical analysis for evaluation performance. The DOI appeared to be effective in conveying the breadth of possibility within our networks. Moreover, the performances are either the synthetic resistivity model or the field resistivity data, both of which have an average of greater than 95%. The inversion results of both architectures are precisely and accurately expressed, containing approximately the ground truth models and thereby also the field observation models. We conclude that these ERT-NET architectures could be one approach to ERT interpretation handling, and we strongly suggest alternatives that promote the geoelectrical method of interpretation.

Faulty-Feeder Detection for Single Phase-to-Ground Faults in Distribution Networks Based on Waveform Encoding and Waveform Segmentation

Faulty feeder detection helps ensure the stability and safety of power grids after single-phase-to-ground (SPG) faults occur in distribution networks. The existing detection techniques identify the faulty feeder by extracting representative fault features, while they fail to show reliable detection performance due to variable fault conditions and complex fault transients. To address these drawbacks, this paper proposes a method based on waveform encoding and waveform segmentation. Since the waveforms have complete fault features in fault signals, it is suitable to recognize the signals on the waveform scale, rather than extracting and fusing several fault features. Firstly, raw sampled zero-sequence voltage (ZSV) and zero-sequence current (ZSC) are processed by using the proposed encoding method, and the ZSV-ZSC image can be generated quickly. Secondly, to learn and understand the waveforms of ZSV and ZSC, a two-path fully convolutional network (FCN) is established to make pixel-wise prediction on the ZSV-ZSC image. Finally, the fault degree of each feeder can be estimated based on the segmented waveform in the ZSV-ZSC image. The performance evaluation is implemented in the NVIDIA Jetson Xavier embedded platform, and the experimental results demonstrate that the proposed method can identify the faulty feeder with high accuracy within 28 ms.

Graph isomorphism U-Net

Graph embedding learning is a fundamental task when dealing with diverse datasets. While encoder–decoder architectures, such as U-Nets, have shown great success in image pixel-wise prediction tasks, applying similar methods to graph data poses challenges due to the lack of natural pooling and up-sampling operations for graphs. Recent methods leverage learnable parameters to extract structural information from neural networks and extend pooling and unpooling to graphs using node features and graph structural information. This paper proposes a novel model called GIUNet (Graph Isomorphism U-Net) for the graph classification task. The proposed Graph U-Net structure is based on graph isomorphism convolution while using a comprehensive pqPooling layer. The pqPooling layer in our approach effectively combines node features and graph structure information during the graph down-sampling stage. To incorporate graph structure information, we utilize both the spectral representation and node centrality measurements. Node centrality measurements capture various structural aspects of nodes in the graph, while the spectral representation helps us focus on the informative low-frequency components of the graph structure. Through ablation studies, we have demonstrated that leveraging the GIUNet model leads to significant improvements compared to state-of-the-art methods across multiple benchmark datasets.

A Robust Pixel-Wise Prediction Network With Applications to Industrial Robotic Grasping

Accurate object detection and 6D pose estimation are the key technologies in robotic grasping applications, where efficiency and robustness are the two most desirable goals. Especially for textureless industrial parts, it is difficult for most existing methods to extract robust image features from cluttered scenarios with heavy occlusion. To address this challenge, we propose a novel pixel-wise prediction strategy using local features to infer global information based on the inherent local-global relations of rigid objects. This strategy is robust to missing or disturbed local information since each pixel has an independent prediction, and the dense prediction manner can mitigate the instability caused by outliers. Accordingly, we first generate dense pixel-wise predictions of the object category, center, and keypoint from image features extracted by an encoder-decoder network. Then, these predictions are used to vote on and identify the keypoint locations of the specific instance object, and finally, the poses are estimated from the keypoints by an uncertainty <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">PnP</i> (Perspective-n-Point) algorithm. Experiments on various scenarios are implemented to illustrate the advantages of our approach on severe industrial scenes, and a robotic grasping platform is constructed to evaluate its application performance.

PL-GNet: Pixel Level Global Network for detection and localization of image forgeries

Unlike most Image Forgery Detection and Localization (IFDL) methods that classify the tampered regions by local patch, the features from the whole image in the spatial and frequency domains are leveraged in this paper to classify each pixel in the image. This paper proposes a high-confidence pixel level global network called PL-GNet to combat real-life image forgery that commonly involves different types and visually looks particularly realistic. There are three building blocks in our end-to-end PL-GNet framework: (1) An Encoding net allows us to extract the global features and generate the high-quality feature maps which indicate possible tampered regions. The newly designed first layer and backbone network architecture based on atrous convolution in Encoding net are adopted to capture the changes of pixel relationships and extract rich multi-scale spatial information. (2) A Long Short Term Memory (LSTM) network based on co-occurrence matrix is designed to capture the tampering traces and the discriminative features between manipulated and non-manipulated regions. (3) A Decoding net that incorporates the output of Encoding net and LSTM network learns the mapping from low-resolution feature maps to pixel-wise prediction masks. Furthermore, we conduct a series of ablation experiments to optimize the design of the Encoding network systematically. Extensive experiments on the six challenging datasets demonstrate that our PL-GNet outperforms each subnetwork and consistently achieves state-of-the-art performance compared to alternative methods over three evaluation metrics.

Haar wavelet downsampling: A simple but effective downsampling module for semantic segmentation

Downsampling operations such as max pooling or strided convolution are ubiquitously utilized in Convolutional Neural Networks (CNNs) to aggregate local features, enlarge receptive field, and minimize computational overhead. However, for a semantic segmentation task, pooling features over the local neighbourhood may result in the loss of important spatial information, which is conducive for pixel-wise predictions. To address this issue, we introduce a simple yet effective pooling operation called the Haar Wavelet-based Downsampling (HWD) module. This module can be easily integrated into CNNs to enhance the performance of semantic segmentation models. The core idea of HWD is to apply Haar wavelet transform for reducing the spatial resolution of feature maps while preserving as much information as possible. Furthermore, to investigate the benefits of HWD, we propose a novel metric, named as feature entropy index (FEI), which measures the degree of information uncertainty after downsampling in CNNs. Specifically, the FEI can be used to indicate the ability of downsampling methods to preserve essential information in semantic segmentation. Our comprehensive experiments demonstrate that the proposed HWD module could (1) effectively improve the segmentation performance across different modality image datasets with various CNN architectures, and (2) efficiently reduce information uncertainty compared to the conventional downsampling methods. Our implementation are available at https://github.com/apple1986/HWD.

LeNo: Adversarial Robust Salient Object Detection Networks with Learnable Noise

Pixel-wise prediction with deep neural network has become an effective paradigm for salient object detection (SOD) and achieved remarkable performance. However, very few SOD models are robust against adversarial attacks which are visually imperceptible for human visual attention. The previous work robust saliency (ROSA) shuffles the pre-segmented superpixels and then refines the coarse saliency map by the densely connected conditional random field (CRF). Different from ROSA that rely on various pre- and post-processings, this paper proposes a light-weight Learnable Noise (LeNo) to defend adversarial attacks for SOD models. LeNo preserves accuracy of SOD models on both adversarial and clean images, as well as inference speed. In general, LeNo consists of a simple shallow noise and noise estimation that embedded in the encoder and decoder of arbitrary SOD networks respectively. Inspired by the center prior of human visual attention mechanism, we initialize the shallow noise with a cross-shaped gaussian distribution for better defense against adversarial attacks. Instead of adding additional network components for post-processing, the proposed noise estimation modifies only one channel of the decoder. With the deeply-supervised noise-decoupled training on state-of-the-art RGB and RGB-D SOD networks, LeNo outperforms previous works not only on adversarial images but also on clean images, which contributes stronger robustness for SOD. Our code is available at https://github.com/ssecv/LeNo.

AN END-TO-END DEEP LEARNING WORKFLOW FOR BUILDING SEGMENTATION, BOUNDARY REGULARIZATION AND VECTORIZATION OF BUILDING FOOTPRINTS

Abstract. Automatic building footprint extraction from remote sensing imagery is a widely used method, with deep learning techniques being particularly effective. However, deep learning approaches still require additional post-processing steps due to pixel-wise predictions, that contribute to occluded and geometrically incorrectly segmented buildings. To address this issue, we propose an end-to-end workflow that utilizes binary semantic segmentation, regularization, and vectorization. We implement and assess the performance of four convolutional neural network architectures including U-Net, U-NetFormer, FT-UnetFormer, and DCSwin on the MapAI Precision in Building Segmentation competition. To additionally improve the shape of the predicted buildings we apply regularization on the predictions to assess whether regularization further improves the geometrical shape and improve the prediction accuracy. We aim to produce accurate predictions with regularized boundaries that can prove useful in many cartographic and engineering applications. The regularization and vectorization workflow is further developed into a working QGIS-plugin that can be used to extend the functionality of QGIS. Our aim is to provide an end-to-end workflow for building segmentation, regularization and vectorization.

Annotation-efficient learning for OCT segmentation.

Deep learning has been successfully applied to OCT segmentation. However, for data from different manufacturers and imaging protocols, and for different regions of interest (ROIs), it requires laborious and time-consuming data annotation and training, which is undesirable in many scenarios, such as surgical navigation and multi-center clinical trials. Here we propose an annotation-efficient learning method for OCT segmentation that could significantly reduce annotation costs. Leveraging self-supervised generative learning, we train a Transformer-based model to learn the OCT imagery. Then we connect the trained Transformer-based encoder to a CNN-based decoder, to learn the dense pixel-wise prediction in OCT segmentation. These training phases use open-access data and thus incur no annotation costs, and the pre-trained model can be adapted to different data and ROIs without re-training. Based on the greedy approximation for the k-center problem, we also introduce an algorithm for the selective annotation of the target data. We verified our method on publicly-available and private OCT datasets. Compared to the widely-used U-Net model with 100% training data, our method only requires of the data for achieving the same segmentation accuracy, and it speeds the training up to ∼3.5 times. Furthermore, our proposed method outperforms other potential strategies that could improve annotation efficiency. We think this emphasis on learning efficiency may help improve the intelligence and application penetration of OCT-based technologies.

Challenges and implications of predicting the spatiotemporal distribution of dengue fever outbreak in Chinese Taiwan using remote sensing data and deep learning

ABSTRACT Ongoing climate change has accelerated the outbreak and expansion of climate-sensitive infectious diseases such as dengue fever. Dengue fever will remain a threat until safe and effective vaccines and antiviral drugs have been developed, distributed, and administered on a global scale. By predicting the spatiotemporal distribution of dengue fever outbreaks, we can effectively implement dengue fever prevention and control. Our study aims to predict the spatiotemporal distribution of dengue fever outbreaks in Chinese Taiwan using a U-Net based encoder – decoder model with daily datasets of sea-surface temperature, rainfall, and shortwave radiation from Remote Sensing (RS) instruments and dengue fever case notification data. Although the prediction accuracy of the proposed model was low and the overlapping areas between the ground truth and pixelwise prediction were few, some of the pixels were located nearby the ground truth, suggesting that the application of RS data and deep learning may help to predict the spatiotemporal distribution of dengue fever outbreaks. With further improvements, the deep learning model might effectively learn a small amount of training data for a specific task.

ActFloor-GAN: Activity-Guided Adversarial Networks for Human-Centric Floorplan Design.

We present a novel two-stage approach for automated floorplan design in residential buildings with a given exterior wall boundary. Our approach has the unique advantage of being human-centric, that is, the generated floorplans can be geometrically plausible, as well as topologically reasonable to enhance resident interaction with the environment. From the input boundary, we first synthesize a human-activity map that reflects both the spatial configuration and human-environment interaction in an architectural space. We propose to produce the human-activity map either automatically by a pre-trained generative adversarial network (GAN) model, or semi-automatically by synthesizing it with user manipulation of the furniture. Second, we feed the human-activity map into our deep framework ActFloor-GAN to guide a pixel-wise prediction of room types. We adopt a re-formulated cycle-consistency constraint in ActFloor-GAN to maximize the overall prediction performance, so that we can produce high-quality room layouts that are readily convertible to vectorized floorplans. Experimental results show several benefits of our approach. First, a quantitative comparison with prior methods shows superior performance of leveraging the human-activity map in predicting piecewise room types. Second, a subjective evaluation by architects shows that our results have compelling quality as professionally-designed floorplans and much better than those generated by existing methods in terms of the room layout topology. Last, our approach allows manipulating the furniture placement, considers the human activities in the environment, and enables the incorporation of user-design preferences.

Adaptive Perspective Distillation for Semantic Segmentation.

Strong semantic segmentation models require large backbones to achieve promising performance, making it hard to adapt to real applications where effective real-time algorithms are needed. Knowledge distillation tackles this issue by letting the smaller model (student) produce similar pixel-wise predictions to that of a larger model (teacher). However, the classifier, which can be deemed as the perspective by which models perceive the encoded features for yielding observations (i.e., predictions), is shared by all training samples, fitting a universal feature distribution. Since good generalization to the entire distribution may bring the inferior specification to individual samples with a certain capacity, the shared universal perspective often overlooks details existing in each sample, causing degradation of knowledge distillation. In this paper, we propose Adaptive Perspective Distillation (APD) that creates an adaptive local perspective for each individual training sample. It extracts detailed contextual information from each training sample specifically, mining more details from the teacher and thus achieving better knowledge distillation results on the student. APD has no structural constraints to both teacher and student models, thus generalizing well to different semantic segmentation models. Extensive experiments on Cityscapes, ADE20K, and PASCAL-Context manifest the effectiveness of our proposed APD. Besides, APD can yield favorable performance gain to the models in both object detection and instance segmentation without bells and whistles.

Dense FixMatch: a simple semi-supervised learning method for pixel-wise prediction tasks

We propose Dense FixMatch, a simple method for online semi-supervised learning of dense and structured prediction tasks combining pseudo-labeling and consistency regularization via strong data augmentation. We enable the application of FixMatch in semi-supervised learning problems beyond image classification by adding a matching operation on the pseudo-labels. This allows us to still use the full strength of data augmentation pipelines, including geometric transformations. We evaluate it on semi-supervised semantic segmentation on Cityscapes and Pascal VOC with different percentages of labeled data and ablate design choices and hyper-parameters. Dense FixMatch significantly improves results compared to supervised learning using only labeled data, approaching its performance with 1/4 of the labeled samples.

SDANet: Semantic-Embedded Density Adaptive Network for Moving Vehicle Detection in Satellite Videos.

In satellite videos, moving vehicles are extremely small-sized and densely clustered in vast scenes. Anchor-free detectors offer great potential by predicting the keypoints and boundaries of objects directly. However, for dense small-sized vehicles, most anchor-free detectors miss the dense objects without considering the density distribution. Furthermore, weak appearance features and massive interference in the satellite videos limit the application of anchor-free detectors. To address these problems, a novel semantic-embedded density adaptive network (SDANet) is proposed. In SDANet, the cluster-proposals, including a variable number of objects, and centers are generated parallelly through pixel-wise prediction. Then, a novel density matching algorithm is designed to obtain each object via partitioning the cluster-proposals and matching the corresponding centers hierarchically and recursively. Meanwhile, the isolated cluster-proposals and centers are suppressed. In SDANet, the road is segmented in vast scenes and its semantic features are embedded into the network by weakly supervised learning, which guides the detector to emphasize the regions of interest. By this way, SDANet reduces the false detection caused by massive interference. To alleviate the lack of appearance information on small-sized vehicles, a customized bi-directional conv-RNN module extracts the temporal information from consecutive input frames by aligning the disturbed background. The experimental results on Jilin-1 and SkySat satellite videos demonstrate the effectiveness of SDANet, especially for dense objects.

Road Topology Extraction From Satellite Imagery by Joint Learning of Nodes and Their Connectivity

Road topology extraction from satellite images, which has long been of interest, is an essential task in remote sensing. The graph representation of road networks is one of the most challenging aspects of road topology extraction. Most existing approaches cast road extraction as binary segmentation and then use post-processing, such as skeletonization, to infer networks from pixel-wise prediction. In our work, we believe that a road network can be represented by an undirected graph denoted as G=(V,E), where V and E represent the set of road nodes and the set of edges between nodes, respectively. Thus, to construct the road topology, we propose NodeConnect, a new method of extracting nodes for a road network and inferring the connectivity between nodes. A convolutional neural network is jointly trained to predict the nodes and connectivity map for nodes, and the edges between nodes are inferred from the connectivity map. We compare our approach with several segmentation methods on the DeepGlobe and RoadTracer datasets. The experiments show that our approach achieves state-of-the-art performance in terms of pixel-based metrics and topological precision and recall.