Object Segmentation Tasks Research Articles

DGANet: A Dilated Graph Attention-Based Network for Local Feature Extraction on 3D Point Clouds

Feature extraction on point clouds is an essential task when analyzing and processing point clouds of 3D scenes. However, there still remains a challenge to adequately exploit local fine-grained features on point cloud data due to its irregular and unordered structure in a 3D space. To alleviate this problem, a Dilated Graph Attention-based Network (DGANet) with a certain feature for learning ability is proposed. Specifically, we first build a local dilated graph-like region for each input point to establish the long-range spatial correlation towards its corresponding neighbors, which allows the proposed network to access a wider range of geometric information of local points with their long-range dependencies. Moreover, by integrating the dilated graph attention module (DGAM) implemented by a novel offset–attention mechanism, the proposed network promises to highlight the differing importance on each edge of the constructed local graph to uniquely learn the discrepancy feature of geometric attributes between the connected point pairs. Finally, all the learned edge attention features are further aggregated, allowing the most significant geometric feature representation of local regions by the graph–attention pooling to fully extract local detailed features for each point. The validation experiments using two challenging benchmark datasets demonstrate the effectiveness and powerful generation ability of our proposed DGANet in both 3D object classification and segmentation tasks.

Cognitive vision inspired object segmentation metric and loss function

Object segmentation (OS) technology is a research hotspot in computer vision, and it has a wide range of applications in many fields. Cognitive vision studies have shown that human vision is highly sensitive to both global information and local details in scenes.To this end, we design a novel, efficient, and easy-to-use enhanced-alignment measure ($E_\\xi$) for evaluating the performance of the OS model.$E_\\xi$ combines local pixel values with the image-level mean value, jointly evaluates the image-/pixel-level similarity between a segmentation result and a ground-truth (GT) result.Extensive experiments on the four popular benchmarks via five meta-measures, i.e., application ranking, demoting generic, denying noise, human ranking, andrecognizing GT, show significant relative improvement compared with existing widely-adopted evaluation metrics such as IoU and $F_\\beta$.By using the weighted binary cross-entropy loss, the enhanced-alignment loss, and the weighted IoU loss, we further design a hybrid loss function (Hybrid-$E_{\\rm~loss}$) to guide the network to learn pixel-, object- and image-level features.Qualitative and quantitative results show further improvement in terms of accuracy when using our hybrid loss function in three different OS tasks.

An Inverse Node Graph-Based Method for the Urban Scene Segmentation of 3D Point Clouds

Urban object segmentation and classification tasks are critical data processing steps in scene understanding, intelligent vehicles and 3D high-precision maps. Semantic segmentation of 3D point clouds is the foundational step in object recognition. To identify the intersecting objects and improve the accuracy of classification, this paper proposes a segment-based classification method for 3D point clouds. This method firstly divides points into multi-scale supervoxels and groups them by proposed inverse node graph (IN-Graph) construction, which does not need to define prior information about the node, it divides supervoxels by judging the connection state of edges between them. This method reaches minimum global energy by graph cutting, obtains the structural segments as completely as possible, and retains boundaries at the same time. Then, the random forest classifier is utilized for supervised classification. To deal with the mislabeling of scattered fragments, higher-order CRF with small-label cluster optimization is proposed to refine the classification results. Experiments were carried out on mobile laser scan (MLS) point dataset and terrestrial laser scan (TLS) points dataset, and the results show that overall accuracies of 97.57% and 96.39% were obtained in the two datasets. The boundaries of objects were retained well, and the method achieved a good result in the classification of cars and motorcycles. More experimental analyses have verified the advantages of the proposed method and proved the practicability and versatility of the method.

Physarum Powered Differentiable Linear Programming Layers and Applications

Consider a learning algorithm, which involves an internal call to an optimization routine such as a generalized eigenvalue problem, a cone programming problem or even sorting. Integrating such a method as layers within a trainable deep network in a numerically stable way is not simple – for instance, only recently, strategies have emerged for eigendecomposition and differentiable sorting. We propose an efficient and differentiable solver for general linear programming problems which can be used in a plug and play manner within deep neural networks as a layer. Our development is inspired by a fascinating but not widely used link between dynamics of slime mold (physarum) and mathematical optimization schemes such as steepest descent. We describe our development and demonstrate the use of our solver in a video object segmentation task and meta-learning for few-shot learning. We review the relevant known results and provide a technical analysis describing its applicability for our use cases. Our solver performs comparably with a customized projected gradient descent method on the first task and outperforms the very recently proposed differentiable CVXPY solver on the second task. Experiments show that our solver converges quickly without the need for a feasible initial point. Interestingly, our scheme is easy to implement and can easily serve as layers whenever a learning procedure needs a fast approximate solution to a LP, within a larger network.

Optical Flow Estimation from a Single Motion-blurred Image

In most of computer vision applications, motion blur is regarded as an undesirable artifact. However, it has been shown that motion blur in an image may have practical interests in fundamental computer vision problems. In this work, we propose a novel framework to estimate optical flow from a single motion-blurred image in an end-to-end manner. We design our network with transformer networks to learn globally and locally varying motions from encoded features of a motion-blurred input, and decode left and right frame features without explicit frame supervision. A flow estimator network is then used to estimate optical flow from the decoded features in a coarse-to-fine manner. We qualitatively and quantitatively evaluate our model through a large set of experiments on synthetic and real motion-blur datasets. We also provide in-depth analysis of our model in connection with related approaches to highlight the effectiveness and favorability of our approach. Furthermore, we showcase the applicability of the flow estimated by our method on deblurring and moving object segmentation tasks.

Meta-VOS: Learning to Adapt Online Target-Specific Segmentation.

The task of video object segmentation is a fundamental but challenging problem in the field of computer vision. To deal with large variations in target objects and background clutter, we propose an online adaptive video object segmentation (VOS) framework, named Meta-VOS, that learns to adapt the target-specific segmentation. Meta-VOS builds an online adaptive learning process by exploiting cumulative expertise after searching for confidence patterns across different videos/frames, and then dynamically improves the model learning from two aspects: Meta-seg learner (i.e., module updating) and Meta-seg criterion (i.e., rule of expertise). As our goal is to rapidly determine which patterns best represent the essential characteristics of specific targets in a video, Meta-seg learner is introduced to adaptively learn to update the parameters and hyperparameters of segmentation network in very few gradient descent steps. Furthermore, a Meta-seg criterion of learned expertise, which is constructed to evaluate the Meta-seg learner for the online adaptation of the segmentation network, can confidently online update positive/negative patterns under the guidance of motion cues, object appearances and learned knowledge. Comprehensive evaluations on several benchmark datasets demonstrate the superiority of our proposed Meta-VOS when compared with other state-of-the-art methods applied to the VOS problem.

Capsules for biomedical image segmentation.

Our work expands the use of capsule networks to the task of object segmentation for the first time in the literature. This is made possible via the introduction of locally-constrained routing and transformation matrix sharing, which reduces the parameter/memory burden and allows for the segmentation of objects at large resolutions. To compensate for the loss of global information in constraining the routing, we propose the concept of "deconvolutional" capsules to create a deep encoder-decoder style network, called SegCaps. We extend the masked reconstruction regularization to the task of segmentation and perform thorough ablation experiments on each component of our method. The proposed convolutional-deconvolutional capsule network, SegCaps, shows state-of-the-art results while using a fraction of the parameters of popular segmentation networks. To validate our proposed method, we perform experiments segmenting pathological lungs from clinical and pre-clinical thoracic computed tomography (CT) scans and segmenting muscle and adipose (fat) tissue from magnetic resonance imaging (MRI) scans of human subjects' thighs. Notably, our experiments in lung segmentation represent the largest-scale study in pathological lung segmentation in the literature, where we conduct experiments across five extremely challenging datasets, containing both clinical and pre-clinical subjects, and nearly 2000 computed-tomography scans. Our newly developed segmentation platform outperforms other methods across all datasets while utilizing less than 5% of the parameters in the popular U-Net for biomedical image segmentation. Further, we demonstrate capsules' ability to generalize to unseen handling of rotations/reflections on natural images.

Improved mask R-CNN-based cloud masking method for remote sensing images

ABSTRACT Clouds lead to missing or distorted land-related information in impacted areas in optical remote sensing images. Cloud masking, which labels cloud-contaminated pixels, forms the basis for subsequent image utilization, such as excluding the distorted pixel or filling in the missing area. However, due to the diverse spectral, textural, and shape characteristics of different clouds and complicated combinations with the underlying land surfaces, cloud masking has become a challenge in remote sensing image processing. In recent years, the Mask region-based convolutional neural network (R-CNN) method, which performs instance segmentation from a complex background and generates a pixelwise mask for the object of interest, has been used widely in object segmentation tasks. When the Mask R-CNN method is used for cloud masking, the mask result has certain problems, such as failing to extract uncommon clouds and outputting inaccurate mask boundaries for large clouds. To address these problems, we introduce two strategies, group training and boundary optimization, to improve the Mask R-CNN. For group training, samples are divided into several groups. The samples in the first group are used for the initial training, and the samples in the next group are used for evaluation. Only samples with missing or falsely detected clouds are used for tuning the classifier; then, these processes are repeated until all groups have been used or the detection precision becomes stable. For boundary optimization, a block-by-block mask strategy is adopted to guarantee that clouds with diverse sizes have similar performances. Finally, two open data sets and one data set labelled by ourselves are selected to test the proposed method, and the results demonstrate that our method can produce cloud masks for different cloud types and diverse underlying land surfaces and can achieve high accuracies, thereby providing an effective alternative for cloud masking. Compared with the original Mask R-CNN method, our method improves the average recall, average precision, and intersection over union by 5.88%, 2.4%, and 0.071 in pixel level, respectively, demonstrating the effectiveness of our improvement.

Interactive machine learning for fast and robust cell profiling.

Automated profiling of cell morphology is a powerful tool for inferring cell function. However, this technique retains a high barrier to entry. In particular, configuring image processing parameters for optimal cell profiling is susceptible to cognitive biases and dependent on user experience. Here, we use interactive machine learning to identify the optimum cell profiling configuration that maximises quality of the cell profiling outcome. The process is guided by the user, from whom a rating of the quality of a cell profiling configuration is obtained. We use Bayesian optimisation, an established machine learning algorithm, to learn from this information and automatically recommend the next configuration to examine with the aim of maximising the quality of the processing or analysis. Compared to existing interactive machine learning tools that require domain expertise for per-class or per-pixel annotations, we rely on users’ explicit assessment of output quality of the cell profiling task at hand. We validated our interactive approach against the standard human trial-and-error scheme to optimise an object segmentation task using the standard software CellProfiler. Our toolkit enabled rapid optimisation of an object segmentation pipeline, increasing the quality of object segmentation over a pipeline optimised through trial-and-error. Users also attested to the ease of use and reduced cognitive load enabled by our machine learning strategy over the standard approach. We envision that our interactive machine learning approach can enhance the quality and efficiency of pipeline optimisation to democratise image-based cell profiling.

Space-Time Memory Networks for Video Object Segmentation With User Guidance.

We propose a novel and unified solution for user-guided video object segmentation tasks. In this work, we consider two scenarios of user-guided segmentation: semi-supervised and interactive segmentation. Due to the nature of the problem, available cues - video frame(s) with object masks (or scribbles) - become richer with the intermediate predictions (or additional user inputs). However, the existing methods make it impossible to fully exploit this rich source of information. We resolve the issue by leveraging memory networks and learning to read relevant information from all available sources. In the semi-supervised scenario, the previous frames with object masks form an external memory, and the current frame as the query is segmented using the information in the memory. Similarly, to work with user interactions, the frames that are given user inputs form the memory that guides segmentation. Internally, the query and the memory are densely matched in the feature space, covering all the space-time pixel locations in a feed-forward fashion. The abundant use of the guidance information allows us to better handle challenges such as appearance changes and occlusions. We validate our method on the latest benchmark sets and achieve state-of-the-art performance along with a fast runtime.

Facial Expression Recognition Based on Weighted-Cluster Loss and Deep Transfer Learning Using a Highly Imbalanced Dataset.

Facial expression recognition (FER) is a challenging problem in the fields of pattern recognition and computer vision. The recent success of convolutional neural networks (CNNs) in object detection and object segmentation tasks has shown promise in building an automatic deep CNN-based FER model. However, in real-world scenarios, performance degrades dramatically owing to the great diversity of factors unrelated to facial expressions, and due to a lack of training data and an intrinsic imbalance in the existing facial emotion datasets. To tackle these problems, this paper not only applies deep transfer learning techniques, but also proposes a novel loss function called weighted-cluster loss, which is used during the fine-tuning phase. Specifically, the weighted-cluster loss function simultaneously improves the intra-class compactness and the inter-class separability by learning a class center for each emotion class. It also takes the imbalance in a facial expression dataset into account by giving each emotion class a weight based on its proportion of the total number of images. In addition, a recent, successful deep CNN architecture, pre-trained in the task of face identification with the VGGFace2 database from the Visual Geometry Group at Oxford University, is employed and fine-tuned using the proposed loss function to recognize eight basic facial emotions from the AffectNet database of facial expression, valence, and arousal computing in the wild. Experiments on an AffectNet real-world facial dataset demonstrate that our method outperforms the baseline CNN models that use either weighted-softmax loss or center loss.

Superpixels for image and video processing based on proximity-weighted patch matching

In this paper, a temporal superpixel algorithm using proximity-weighted patch matching (PPM) is proposed to yield temporally consistent superpixels for image and video processing. PPM estimates the motion vector of a superpixel robustly, by considering the patch matching distances of neighboring superpixels as well as the superpixel itself. In each frame, we initialize superpixels by transferring the superpixel labels of the previous frame using PPM motion vectors. Then, we update the superpixel labels of boundary pixels by minimizing a cost function, which is composed of feature distance, compactness, contour, and temporal consistency terms. Finally, we carry out superpixel splitting, merging, and relabeling to regularize superpixel sizes and correct inaccurate labels. Extensive experimental results confirm that the proposed algorithm outperforms the state-of-the-art conventional algorithms significantly. Also, it is demonstrated that the proposed algorithm can be applied to video object segmentation and video saliency detection tasks.

Extra FAT: A Photorealistic Dataset for 6D Object Pose Estimation

We introduce a new image dataset for object detection and 6D pose estimation, named Extra FAT. The dataset consists of 825K photorealistic RGB images with annotations of groundtruth location and rotation for both the virtual camera and the objects. A registered pixel-level object segmentation mask is also provided for object detection and segmentation tasks. The dataset includes 110 different 3D object models. The object models were rendered in five scenes with diverse illumination, reflection, and occlusion conditions.

Graph Moving Object Segmentation

Moving Object Segmentation (MOS) is a fundamental task in computer vision. Due to undesirable variations in the background scene, MOS becomes very challenging for static and moving camera sequences. Several deep learning methods have been proposed for MOS with impressive performance. However, these methods show performance degradation in the presence of unseen videos; and usually, deep learning models require large amounts of data to avoid overfitting. Recently, graph learning has attracted significant attention in many computer vision applications since they provide tools to exploit the geometrical structure of data. In this work, concepts of graph signal processing are introduced for MOS. First, we propose a new algorithm that is composed of segmentation, background initialization, graph construction, unseen sampling, and a semi-supervised learning method inspired by the theory of recovery of graph signals. Second, theoretical developments are introduced, showing one bound for the sample complexity in semi-supervised learning, and two bounds for the condition number of the Sobolev norm. Our algorithm has the advantage of requiring less labeled data than deep learning methods while having competitive results on both static and moving camera videos. Our algorithm is also adapted for Video Object Segmentation (VOS) tasks and is evaluated on six publicly available datasets outperforming several state-of-the-art methods in challenging conditions.

Zero-Shot Video Object Segmentation With Co-Attention Siamese Networks.

We introduce a novel network, called CO-attention siamese network (COSNet), to address the zero-shot video object segmentation task in a holistic fashion. We exploit the inherent correlation among video frames and incorporate a global co-attention mechanism to further improve the state-of-the-art deep learning based solutions that primarily focus on learning discriminative foreground representations over appearance and motion in short-term temporal segments. The co-attention layers in COSNet provide efficient and competent stages for capturing global correlations and scene context by jointly computing and appending co-attention responses into a joint feature space. COSNet is a unified and end-to-end trainable framework where different co-attention variants can be derived for capturing diverse properties of the learned joint feature space. We train COSNet with pairs (or groups) of video frames, and this naturally augments training data and allows increased learning capacity. During the segmentation stage, the co-attention model encodes useful information by processing multiple reference frames together, which is leveraged to infer the frequently reappearing and salient foreground objects better. Our extensive experiments over three large benchmarks demonstrate that COSNet outperforms the current alternatives by a large margin. Our implementations are available at https://github.com/carrierlxk/COSNet.

Bi-Directional Seed Attention Network for Interactive Image Segmentation

In interactive segmentation, the role of seed information provided by the user is significant. A seed is a clue to ease the ambiguity of the problem by making the object segmentation task interactive. However, in most deep network-based works, seed information has been used as an additional channel for input images. In this paper, we propose a novel bi-directional attention module for more actively using seed information. The proposed bi-directional seed attention module (BSA) operates based on the feature map of the segmentation network and the input seed map. Through our attention module, the network feature map is affected by the seed map, while the feature also updates the seed information. As a result, our system concentrates on the seed information and more accurately derives the segmentation result required by the user. We have conducted validation experiments on the four standard benchmark datasets, including SBD, GrabCut, Berkeley, and DAVIS, and achieved the state-of-the-art results.

Flow Adaptive Video Object Segmentation

We tackle the task of semi-supervised video object segmentation, i.e. pixel-level object classification of the images in video sequences using very limited ground truth training data of its corresponding video. We present FLow Adaptive Video Object Segmentation, an efficient pipeline based on a novel online adaptation algorithm that utilizes optical flow, capable of tracking objects effectively throughout videos. Comparing with most of the recent deep learning based approaches that trade efficiency for accuracy, we provide extensive complexity analysis and additionally demonstrate that FLAVOS is natural for real world applications by introducing an interactive pipeline that enables the user to provide feedback for online training. Our method achieves state-of-the-art accuracy on three challenging benchmark datasets and nearly ground-truth level segmentation results with interactive user feedback.

PointNGCNN: Deep convolutional networks on 3D point clouds with neighborhood graph filters

Despite great success of deep neural networks for 2D vision tasks, point clouds, unlike 2D images, cannot be directly applied to traditional convolutional neural networks because of irregularities in the form of data. In this paper, we develop a novel end-to-end deep learning network called PointNGCNN that can consume point clouds for 3D object recognition and segmentation tasks. In order to extract the neighborhood geometric features, we propose to construct a neighborhood graph that reflects the relationship between the neighborhood points of each point and then use the Chebyshev polynomials as the neighborhood graph filters. Further, we put the feature matrix and Laplacian matrix of each neighborhood into the network and use the max pooling operation to get the features of each center. Experimental results on benchmark datasets demonstrate that PointNGCNN has achieved good performance in the recognition and segmentation tasks.

Video Object Segmentation based on improved OSVOS

Video object segmentation is a mainstream branch of current image processing direction. How to achieve deep learning from complete supervision to unsupervised is the key problem that people are trying to solve. In this process, One-Shot Video Object Segmentation (OSVOS) can successfully tackle the task of semi-supervised video object segmentation. It transfers the general semantic information learned on ImageNet to the foreground segmentation task, and then learns the mapping of a single annotated object in sequence. In this paper, based on the concept of OSVOS, an improved neural network structure with dilated convolution, multi-scale convolution fusion and skip layer is proposed. Dilated convolution can increase the size of the receiving field. Multi-scale convolution can obtain multi-scale feature maps and fuse them. Skip layers can transfer feature information from low layer to upper layer. All of them help to improve the final accuracy. The experimental results show that all indicators has been improved.

Effective online refinement for video object segmentation

In this paper, we propose a novel framework, which deeply explores the motion cue and the online fine-tuning strategy to tackle the task of semi-supervised video object segmentation. First, in order to filter out the irrelevant background regions in the initial segmentation results, which are generated by an existing semi-supervised segmentation model, a motion based background suppression method is exploited to obtain the purified segmentation results. Second, a set of key frames with high-quality segmentation results are selected based on several metrics of segmentation quality in the purified segmentation results. Finally, the selected key frames are combined with the manually annotated first frame to efficiently retrain the segmentation model online, so as to obtain more accurate segmentation results. Our experimental results on two challenging datasets demonstrate that the proposed framework achieves the state-of-the-art performance.