Unique correct correspondence cannot be obtained only by use of gray correlation technique, which describes gray similar degree of feature points between the left and right images too unilaterally. The gray correlation technique is adopted to extract gray correlation peaks as a coarse matching set called multi-peak set. The disparity gradient limited constraint is utilized to optimize the multi-peak set. Unique match will be obtained by calculating the correlation of hybrid matrices consisting of reference differences and disparities from the multi-peak set. Two of the known corresponding points in the left and right images, respectively, are set as a pair of reference points to determine search direction and search scope at first. After the unique correspondence is obtained by calculating the correlation of the hybrid matrices from the multi-peak set, the obtained match is regarded as a new reference point till all feature points in the left (or right) image have been processed. Experimental results proved that the proposed algorithm was feasible and accurate.