We present a quantum algorithm for finding the minimum of a function based on multistep quantum computation, and apply the algorithm for solving optimization problems with continuous variables. We construct the state space of the problem by discretizing the variables of the problem, and divide the state space according to the function values of the vectors of the state space. By comparing the function values of the vectors with a series of threshold values in decreasing order, we construct a sequence of Hamiltonians where the search space of a Hamiltonian is nested in that of the previous one. By applying a multistep quantum computation process for finding the ground state of the last Hamiltonian, the optimal vector of the state space of the problem is located in a small search space and can be determined efficiently. One of the most difficult problems in optimization algorithms is that a trial vector is trapped in a deep local minimum while the global minimum is missed, this problem can be alleviated in our algorithm and the run time is proportional to the number of the steps of the algorithm, provided that the reduction rate of the search spaces is polynomial large. We discuss the implementation of the algorithm and test the algorithm for some test functions.