A stereo recording can be considered to consist of primary and ambient components in both left and right channels. Decomposing a stereo recording into primary and ambient components is a crucial step in upmixing to retain its spatial information. The state-of-the-art algorithm to carry out the primary ambient extraction (PAE), namely the ambient phase estimation with a sparsity constraint (APES), utilized the sparsity of the primary components in the time–frequency domain to formulate the ambient phase estimation as a non-convex optimization problem. Hence, the discrete searching method was adopted, resulting in a computationally complicated solving process. In this paper, a fast non-uniform searching strategy is proposed to improve the efficiency of the APES, resulting in the ambient phase estimation with a sparsity constraint and non-uniform searching (APEN). Objective and subjective results validate that the extraction error of the APEN is almost the same as that of the APES, while the algorithm complexity of the APEN is reduced to one third that of the APES.