Natural Image Matting Research Articles

Matte anything: Interactive natural image matting with segment anything model

Natural image matting algorithms aim to predict the transparency map (alpha-matte) with the trimap guidance. However, the production of trimap often requires significant labor, which limits the widespread application of matting algorithms on a large scale. To address the issue, we propose Matte Anything model (MatAny), an interactive natural image matting model that could produce high-quality alpha-matte with various simple hints. The key insight of MatAny is to generate pseudo trimap automatically with contour and transparency prediction. In our work, we leverage vision foundation models to enhance the performance of natural image matting. Specifically, we use the segment anything model to predict high-quality contour with user interaction and an open-vocabulary detector to predict the transparency of any object. Subsequently, a pre-trained image matting model generates alpha mattes with pseudo trimaps. MatAny is the interactive matting algorithm with the most supported interaction methods and the best performance to date. It consists of orthogonal vision models without any additional training. We evaluate the performance of MatAny against several previous image matting algorithms. MatAny has 58.3% improvement on MSE and 40.6% improvement on SAD compared to the previous image matting methods with simple guidance, achieving new state-of-the-art (SOTA) performance. The source codes and pre-trained models are available at https://github.com/hustvl/Matte-Anything.

Color subspace exploring for natural image matting

AbstractDeep neural networks have seen a surge of successful methods in natural image matting. However, the overlap of foreground and background color distributions in an image is still troubling in matting. It is observed that the three color channels contain different contrast information of an image: some color channels may provide clearer contrast information for separating the foreground from the image, while the foreground and background color distributions in other channels may heavily overlap, resulting in blurred foreground‐background boundaries. Motivated by this observation, the Color Subspace Exploring Network (CSEMat) is proposed to extract the foreground object from an image by exploring high‐contrast appearance information in individual color spaces. Specifically, a 4‐branch encoder is constructed, with one branch for the RGB image and three branches for subdividing the color space. Each color channel is individually processed by a sub‐encoder. Additionally, the trimap‐based color information aggregation module (CIA) is introduced to integrate the feature maps from the independent sub‐encoders, facilitating the transfer of optimized features to the decoder. Extensive experiments demonstrate that the proposed CSEMat achieves favorable performance on publicly available matting datasets.

Effective Local-Global Transformer for Natural Image Matting

Learning-based matting methods have been dominated by convolution neural networks for a long time. These methods mainly propagate the alpha matte according to the similarity between unknown and known regions. However, correlations between pixels in unknown and known regions are limited due to the insufficient receptive fields of common convolution neural networks, which leads to inaccurate estimation for pixels in unknown regions that are far away from known regions. In this paper, we propose an Effective Local-Global Transformer for natural image matting (ELGT-Matting), which can further expand receptive fields to establish a wide range of correlations between unknown and known regions. The kernel module is the effective local-global transformer block, and each block consists of two modules: 1) A Window-Level Global MSA (Multi-head Self-Attention) module, which learns global context features among windows. 2) A Local-Global Window MSA, which combines coarse global context features and corresponding fine local window features to help local window self-attention capture both local and context information. Experiments demonstrate that our ELGT-Matting performs outstandingly against other competitive approaches on Composition-1K, Distinctions-646, and real-world AIM-500 datasets. In particular, we achieve a new SOTA result on Composition-1K with MSE 0.00374.

Multi-Criterion Sampling Matting Algorithm via Gaussian Process.

Natural image matting is an essential technique for image processing that enables various applications, such as image synthesis, video editing, and target tracking. However, the existing image matting methods may fail to produce satisfactory results when computing resources are limited. Sampling-based methods can reduce the dimensionality of the decision space and, therefore, reduce computational resources by employing different sampling strategies. While these approaches reduce computational consumption, they may miss an optimal pixel pair when the number of available high-quality pixel pairs is limited. To address this shortcoming, we propose a novel multi-criterion sampling strategy that avoids missing high-quality pixel pairs by incorporating multi-range pixel pair sampling and a high-quality sample selection method. This strategy is employed to develop a multi-criterion matting algorithm via Gaussian process, which searches for the optimal pixel pair by using the Gaussian process fitting model instead of solving the original pixel pair objective function. The experimental results demonstrate that our proposed algorithm outperformed other methods, even with 1% computing resources, and achieved alpha matte results comparable to those yielded by the state-of-the-art optimization algorithms.

Bridging Composite and Real: Towards End-to-End Deep Image Matting

Extracting accurate foregrounds from natural images benefits many downstream applications such as film production and augmented reality. However, the furry characteristics and various appearance of the foregrounds, e.g., animal and portrait, challenge existing matting methods, which usually require extra user inputs such as trimap or scribbles. To resolve these problems, we study the distinct roles of semantics and details for image matting and decompose the task into two parallel sub-tasks: high-level semantic segmentation and low-level details matting. Specifically, we propose a novel Glance and Focus Matting network (GFM), which employs a shared encoder and two separate decoders to learn both tasks in a collaborative manner for end-to-end natural image matting. Besides, due to the limitation of available natural images in the matting task, previous methods typically adopt composite images for training and evaluation, which result in limited generalization ability on real-world images. In this paper, we investigate the domain gap issue between composite images and real-world images systematically by conducting comprehensive analyses of various discrepancies between the foreground and background images. We find that a carefully designed composition route RSSN that aims to reduce the discrepancies can lead to a better model with remarkable generalization ability. Furthermore, we provide a benchmark containing 2,000 high-resolution real-world animal images and 10,000 portrait images along with their manually labeled alpha mattes to serve as a test bed for evaluating matting model’s generalization ability on real-world images. Comprehensive empirical studies have demonstrated that GFM outperforms state-of-the-art methods and effectively reduces the generalization error. The code and the datasets will be released at https://github.com/JizhiziLi/GFM .

Attention Transfer Network for Nature Image Matting

Natural image matting is an important problem that widely applied in computer vision and graphics. Recent deep learning matting approaches have made an impressive process in both accuracy and efficiency. However, there are still two fundamental problems remain largely unsolved: 1) accurately separating an object from the image with similar foreground and background color or lots of details; 2) exactly extracting an object with fine structures from complex background. In this paper, we propose an attention transfer network (ATNet) to overcome these challenges. Specifically, we firstly design a feature attention block to effectively distinguish the foreground object from the color-similar regions by activating foreground-related features as well as suppressing others. Then, we introduce a scale transfer block to magnify the feature maps without adding extra information. By integrating the above blocks into an attention transfer module, we effectively reduce the artificial content in results and decrease the computational complexity. Besides, we use a perceptual loss to measure the difference between the feature representations of the predictions and the ground-truths. It can further capture the high-frequency details of the image, and consequently, optimize the fine structures of the object. Extensive experiments on two publicly common datasets ( i.e. , Composition-1k matting dataset, and www.alphamatting.com dataset) show that the proposed ATNet obtains significant improvements over the previous methods. The source code and compiled models have been made publicly available at https://github.com/ailsaim/ATNet .

Natural Image Matting via Guided Contextual Attention

Over the last few years, deep learning based approaches have achieved outstanding improvements in natural image matting. Many of these methods can generate visually plausible alpha estimations, but typically yield blurry structures or textures in the semitransparent area. This is due to the local ambiguity of transparent objects. One possible solution is to leverage the far-surrounding information to estimate the local opacity. Traditional affinity-based methods often suffer from the high computational complexity, which are not suitable for high resolution alpha estimation. Inspired by affinity-based method and the successes of contextual attention in inpainting, we develop a novel end-to-end approach for natural image matting with a guided contextual attention module, which is specifically designed for image matting. Guided contextual attention module directly propagates high-level opacity information globally based on the learned low-level affinity. The proposed method can mimic information flow of affinity-based methods and utilize rich features learned by deep neural networks simultaneously. Experiment results on Composition-1k testing set and alphamatting.com benchmark dataset demonstrate that our method outperforms state-of-the-art approaches in natural image matting. Code and models are available at https://github.com/Yaoyi-Li/GCA-Matting.

Deep Convolutional Neural Network for Natural Image Matting Using Initial Alpha Mattes.

We propose a deep convolutional neural network (CNN) method for natural image matting. Our method takes multiple initial alpha mattes of the previous methods and normalized RGB color images as inputs, and directly learns an end-to-end mapping between the inputs and reconstructed alpha mattes. Among the various existing methods, we focus on using two simple methods as initial alpha mattes: the closed-form matting and KNN matting. They are complementary to each other in terms of local and nonlocal principles. A major benefit of our method is that it can "recognize" different local image structures and then combine the results of local (closed-form matting) and nonlocal (KNN matting) mattings effectively to achieve higher quality alpha mattes than both of the inputs. Furthermore, we verify extendability of the proposed network to different combinations of initial alpha mattes from more advanced techniques such as KL divergence matting and information-flow matting. On the top of deep CNN matting, we build an RGB guided JPEG artifacts removal network to handle JPEG block artifacts in alpha matting. Extensive experiments demonstrate that our proposed deep CNN matting produces visually and quantitatively high-quality alpha mattes. We perform deeper experiments including studies to evaluate the importance of balancing training data and to measure the effects of initial alpha mattes and also consider results from variant versions of the proposed network to analyze our proposed DCNN matting. In addition, our method achieved high ranking in the public alpha matting evaluation dataset in terms of the sum of absolute differences, mean squared errors, and gradient errors. Also, our RGB guided JPEG artifacts removal network restores the damaged alpha mattes from compressed images in JPEG format.

A Three-Stage Matting Method

Natural image matting is an important image processing task. How to leverage the advantages of both sampling matting and propagation matting is a challenge issue. In this paper, we propose a novel sampling-propagation matting method. First, in an overall framework, we propose a three-stage method for sampling-propagation matting, in which the sampling matting stage and propagation matting stage are bridged by a new stage (stage 2). Second, in the sampling matting stage (stage 1), a new gradient sampling matting is presented to cover more diversified samples, and a new equation is proposed to calculate the impact of sample-pair overlap. Third, in bridge stage (stage 2), we propose a judgment criterion to distinguish each pixel of matte after stage 1. For pixels that fail to meet the criterion, we propose an automatic labeling method. Fourth, in the propagation stage (stage 3), we discriminatingly process non-labeled pixels and labeled pixels with separate weight coefficients. The non-labeled pixels are smoothed by propagation, and the labeled pixels are solved by propagation matting. Finally, the proposed method is compared with other methods on public available benchmark. The results show that our method outperforms many other methods and achieves a good ranking.

KNN Matting

This paper proposes to apply the nonlocal principle to general alpha matting for the simultaneous extraction of multiple image layers; each layer may have disjoint as well as coherent segments typical of foreground mattes in natural image matting. The estimated alphas also satisfy the summation constraint. As in nonlocal matting, our approach does not assume the local color-line model and does not require sophisticated sampling or learning strategies. On the other hand, our matting method generalizes well to any color or feature space in any dimension, any number of alphas and layers at a pixel beyond two, and comes with an arguably simpler implementation, which we have made publicly available. Our matting technique, aptly called KNN matting, capitalizes on the nonlocal principle by using $(K)$ nearest neighbors (KNN) in matching nonlocal neighborhoods, and contributes a simple and fast algorithm that produces competitive results with sparse user markups. KNN matting has a closed-form solution that can leverage the preconditioned conjugate gradient method to produce an efficient implementation. Experimental evaluation on benchmark datasets indicates that our matting results are comparable to or of higher quality than state-of-the-art methods requiring more involved implementation. In this paper, we take the nonlocal principle beyond alpha estimation and extract overlapping image layers using the same Laplacian framework. Given the alpha value, our closed form solution can be elegantly generalized to solve the multilayer extraction problem. We perform qualitative and quantitative comparisons to demonstrate the accuracy of the extracted image layers.

Material matting

Despite the widespread use of measured real-world materials, intuitive tools for editing measured reflectance datasets are still lacking. We present a solution inspired by natural image matting and texture synthesis to the material matting problem, which allows separating a measured spatially-varying material into simpler foreground and background component materials and a corresponding opacity map. We approach this problem in the context of Bayesian statistics and introduce a new prior on materials that favors those with highly self-similar stochastic structure. We describe a prototype system that iteratively performs these separations based on small sets of user scribbles and demonstrate multiple separations and edits.

Automatic Extraction Method for License Plate Character Blurred by Translucent Object

Characters on license plate, which is obscured by water or mist, is hard to be recognized through classical license plate recognition technology. In order to get higher recognition rate of blurred character in practical license plate recognition system, an approach to separate characters from a blurred image of vehicle license plate is proposed in this paper. At first, the upper, lower, left and right boundaries of blurred character are assumed to be foreground sample; the broken boundary of character is considered as background sample. At the position of background white scribbles are drawn; in contrast, midpoints of the lines along foreground are painted with black scribbles. Then a scribble image is generated automatically. Furthermore, closed form solution to natural image matting is adopted to obtain the background that is uncovered license plate according to the scribble image. The experiments show that the new algorithm is applicable and helpful to get higher recognition rate of blurred license plate.

Fractional Stereo Matching Using Expectation-Maximization

In our fractional stereo matching problem, a foreground object with a fractional boundary is blended with a background scene using unknown transparencies. Due to the spatially varying disparities in different layers, one foreground pixel may be blended with different background pixels in stereo images, making the color constancy commonly assumed in traditional stereo matching not hold any more. To tackle this problem, in this paper, we introduce a probabilistic framework constraining the matching of pixel colors, disparities, and alpha values in different layers, and propose an automatic optimization method to solve a Maximizing a Posterior (MAP) problem using Expectation-Maximization (EM), given only a short-baseline stereo input image pair. Our method encodes the effect of background occlusion by layer blending without requiring a special detection process. The alpha computation process in our unified framework can be regarded as a new approach by natural image matting, which handles appropriately the situation when the background color is similar to that of the foreground object. We demonstrate the efficacy of our method by experimenting with challenging stereo images and making comparisons with state-of-the-art methods.

Natural Image Matting with Membership Propagation

We present a semi-supervised technique of object extraction for natural image matting. At first, we present a novel unsupervised graph-spectral algorithm for extraction of homogeneous regions in an image. We next derive a semi-supervised scheme from this unsupervised algorithm. In our method, it is sufficient for users to draw strokes only in one of object and background regions. The semi-supervised optimization problem is solved with an iterative method where memberships are propagated from strokes to their surroundings. We suggest a guideline for placement of strokes by exploiting the same iterative solution process in the unsupervised algorithm. We project the color vectors with the linear discriminant analysis to improve the color discriminability and speed up the convergence of the iterative method. Performance of the proposed method is examined for some images and the results are compared with other methods and ground truth mattes.

A Closed-Form Solution to Natural Image Matting

Interactive digital matting, the process of extracting a foreground object from an image based on limited user input, is an important task in image and video editing. From a computer vision perspective, this task is extremely challenging because it is massively ill-posed -- at each pixel we must estimate the foreground and the background colors, as well as the foreground opacity ("alpha matte") from a single color measurement. Current approaches either restrict the estimation to a small part of the image, estimating foreground and background colors based on nearby pixels where they are known, or perform iterative nonlinear estimation by alternating foreground and background color estimation with alpha estimation. In this paper we present a closed-form solution to natural image matting. We derive a cost function from local smoothness assumptions on foreground and background colors, and show that in the resulting expression it is possible to analytically eliminate the foreground and background colors to obtain a quadratic cost function in alpha. This allows us to find the globally optimal alpha matte by solving a sparse linear system of equations. Furthermore, the closed-form formula allows us to predict the properties of the solution by analyzing the eigenvectors of a sparse matrix, closely related to matrices used in spectral image segmentation algorithms. We show that high quality mattes for natural images may be obtained from a small amount of user input.

Least square image matting

This paper addresses the well-known problem of natural image matting. Most of the previous matting algorithms require the user to define the tri-map, which is an inconvenient work and sometimes a b...

Fast natural image matting in perceptual color space

This paper proposes a fast natural image matting method in perceptual color space. Natural image matting is usually composed of three steps: region segmenting, color estimating and alpha estimating. Our matting approach uses a practical model to estimate the foreground and background color of a given pixel in unknown regions. It avoids complex computing and decreases computational cost significantly. A new alpha estimating method in perceptual color space is introduced to compute the alpha value correctly and effectively. We separate the chroma and intensity information of a color and emphasize the more significant one. Our method works well on different perceptual color spaces, extracts foreground objects much faster, and produces modestly better mattes than the Bayesian approach.

Video matting of complex scenes

This paper describes a new framework for video matting, the process of pulling a high-quality alpha matte and foreground from a video sequence. The framework builds upon techniques in natural image matting, optical flow computation, and background estimation. User interaction is comprised of garbage matte specification if background estimation is needed, and hand-drawn keyframe segmentations into "foreground," "background" and "unknown". The segmentations, called trimaps, are interpolated across the video volume using forward and backward optical flow. Competing flow estimates are combined based on information about where flow is likely to be accurate. A Bayesian matting technique uses the flowed trimaps to yield high-quality mattes of moving foreground elements with complex boundaries filmed by a moving camera. A novel technique for smoke matte extraction is also demonstrated.