Inverse design method is widely used in photonic device design. To accelerate the speed to find the possible solutions in design space, optimization method with high efficiency to exhaust the parameter space is pivotal. In this paper, we propose a gradient-oriented binary search method for photonic device design. It utilizes the gradient of refractive index change to the figure of merit to guide the flipping process between binary materials. This method can accelerate the optimization convergence speed compared with direct binary search method, while maintain the advantages of no additional binarization procedure and well critical dimension constraints. Based on this design methodology, we design a bimodal interferometer on silicon photonics platform, which can generate interference pattern of TE1 and TE2 mode using fundamental mode as input. The device layout is 4 μm × 8 μm. To verify the universality of the method, we design a TE1-TE0 mode converter and 3-dB optical power splitter. Results also verifies that the optimize speed of this method is much faster than direct binary search method. And the convergence iteration steps would not be significantly increased with the increase of unit cells. Our approach provides a promising supplement to the direct binary method, and can be implemented in a variety of device design.